Thursday, January 9, 2020

AUTOMERGE

I started writing this post four and one half years ago. I'm finally getting around to finishing it because I had to use it in real life this week.

I ran into what I thought was strange behavior in ASO the other day. I'd submit data to a cell, then submit a zero. I'd see a zero. Then I'd submit data to a cell, then submit a zero. I'd see a #MISSING. I'd repeat this over and over again. The first time I'd get a zero, the second time I'd get a #MISSING. At least that was consistent!

I had a hunch it had to do with ASO data slices and the merge process. In 11.1.2.4 the default ASO behavior is for slices to be automatically merged after some number of slices have been created. The two essbase.cfg settings that control this are AUTOMERGE and AUTOMERGEMAXSLICENUMBER.

So the first send of a number created slice 1, the send of the zero created slice 2, the send of the next number created slice 3, the send of the next zero created slice 4. Four happens to be the default value for the AUTOMERGEMAXSLICENUMBER setting. ALWAYS happens to be the default value for the AUTOMERGE setting. So when that fourth slice was created, it triggered Essbase to automatically merge the slices back together.

The documentation states that the merge happens after the slice number is exceeded, so it should happen on the fifth load of data. I observed that it was happening on the exact number, so on the fourth load of data.

The one thing you can't control here is that when the merge happens it will turn your zeros into #MISSINGs. In most cases this is fine, but in certain circumstances when you need an actual zero, it could be a problem. Merging the data slices using MaxL allows you to control whether or not the zeros are removed.

alter database ASOsamp.Sample merge all data remove_zero_cells;
I'm curious to know if anyone has run into any problems with this setting causing performance problems. Depending upon how many users are sending data to the cube I would imagine running so many merges could be an issue. Then again, it might help since you reduce query cost after merging slices.

Friday, September 13, 2019

MDX Insert Intermission

I have not created any posts about MDX Insert lately as I have hit a snag with my research on the CUSTOMCALCANDALLOCTHRUINSERT Essbase config file setting. This setting, in theory, will take an existing ASO procedural calc and automatically run it as an MDX Insert. This would be great for the customers who have a large number of ASO procedural calcs. I get the impression that once Essbase 19c comes out that this will work. So I'll wait until I have better information before doing a full write-up on the topic.

In the meantime I can report some really great news about MDX Insert and my hope for it becoming the next "big thing" for Essbase.

I was working with a colleague on an ASO procedural calc. He was using 11.1.2.3.500 and ran into the "query exceeds 2^64" error which kills the calc. This was due to the large number of combinations in the POV.

The same calc in 11.1.2.4.031 doesn't give an error but it also never finishes. I think a lot of people were turned off to ASO procedural calcs due to this issue.

The great news is that the same calc that died previously, when translated to MDX Insert, took a grand total of two seconds to complete in OAC. Two frikkin' seconds!

I'm seeking other outlines to try this on. If you've got one I can use for testing, drop me a line.

Also, I'll be at Oracle Open World 19 in San Francisco next week. I hope to see some of you there.

Tuesday, August 6, 2019

MDX Insert Part Three



In Part Two of this series I showed how to perform a simple MDX Insert, copying a single cell of data in an ASO cube. In this post I will show how to create a slightly more complicated calculation.

Keep in mind that you need to be using Essbase in OAC for this to work. It will not work in Essbase 11.1.2.4 or prior.

WITH

I'm going to use the MDX 'With' statement in order to create a calculated member. The With statement comes first in an MDX script. You can use it to define an Essbase member on the fly. You can then use that member in other calculations or in the output of an MDX query. To keep things simple and easy to reproduce for those of you following along at home, I will recreate the Variance member in ASOSamp.Basic.

First I added a member in the outline called Stored Variance.



Next I created my MDX statement. The 'With' statement comes first. In this case I called my member on the fly _Variance. I put an underscore in front of members on the fly so they won't conflict with any Essbase members and so that I can quickly see what's being created dynamically in MDX. Of course, if you happen to use underscores to begin member names in your Essbase outline (heaven help you and) find a different method.

The syntax here is hopefully straightforward. You define the member name AS some formula which goes in single quotes. Here I am subtracting [Prev Year] from [Curr Year].





Next comes the INSERT statement. Remember from part two that the syntax is "(source tuple)" TO "(target tuple)".



Then the SELECT statement is added surrounded by parenthesis. In this case I'm putting the _Variance member on the columns and then the Crossjoin of all the level zero members from the remaining dimensions on the rows. Notice that I don't need to bother leaving out shared or dynamic members.



Once I validate and save the script, I can run the job. To do that I go to the jobs screen in the Jet UI.




Then select New Job -> Run MDX.




Finally, I select the MDX script I want to run.



This calculation took 6.9 seconds to run with the standard ASOSamp.Basic data.


Why?

Why might I want to run a calculation like this? In this case it doesn't make a lot of sense. ASO should be able to handle a simple variance calculation with little effort. But what about a rate times volume calculation? Or a foreign currency calculation? Those get much trickier -- see why here.

The Old Way

You could run this calculation with the Execute Allocation MaxL statement. Here's the statement:
 execute allocation process on database ASOSamp.Basic with  
 pov "CROSSJOIN(Except([Measures].LEVELS(0).MEMBERS,{[Avg Units/Transaction],[% of Total]}),  
     CROSSJOIN(Descendants([MTD],LEVELS([Time],0)),    
     CROSSJOIN([Transaction Type].LEVELS(0).MEMBERS,   
     CROSSJOIN([Payment Type].LEVELS(0).MEMBERS,   
     CROSSJOIN([Promotions].LEVELS(0).MEMBERS,   
     CROSSJOIN([Age].LEVELS(0).MEMBERS,   
     CROSSJOIN([Income Level].LEVELS(0).MEMBERS,   
     CROSSJOIN([Products].LEVELS(0).MEMBERS,   
     CROSSJOIN([Stores].LEVELS(0).MEMBERS,[Geography].LEVELS(0).MEMBERS)))))))))"  
 amount "([Curr Year]-[Prev Year])"  
 target ""  
 range "{[Stored Variance]}"  
 spread;  

Notice that I had to add in an Except() statement to remove members with formulas. I also had to change the way I was pulling the Time dimension members because they too had some formulas. When I ran the script I got these warnings:

 [DBNAME: Basic] The argument [POV] ignores the duplicate member [Flat Panel]  
 [DBNAME: Basic] The argument [POV] ignores the duplicate member [HDTV]  
 [DBNAME: Basic] The argument [POV] ignores the duplicate member [Digital Recorders]  
 [DBNAME: Basic] The argument [POV] ignores the duplicate member [Notebooks]  

So I probably should have removed those shared members from the POV as well. What a pain...

This script took 7.9 seconds to run. I understand that 1 second is not a lot of time but in this case the MDX Insert was about 15% faster than the Execute Allocation. I've always seen performance improvement for MDX Insert when comparing the two methods. Over the course of many longer calculations, the performance improvement can be significant.



NON EMPTY

In case you're wondering, I did try adding a NON EMPTY directive before the rows set and it yielded no performance improvement.
 

Appendix

MDX Script for easy copy and paste:
 WITH  
 MEMBER Years.[_Variance] AS  
   '[Curr Year]-[Prev Year]'  
 INSERT  
   "(Years.[_Variance])" TO "(Years.[Stored Variance])"  
 INTO ASOSamp.Basic FROM   
 (  
   SELECT  
    {Years.[_Variance]} ON COLUMNS,  
    {CROSSJOIN([Measures].LEVELS(0).MEMBERS,  
     CROSSJOIN([Time].LEVELS(0).MEMBERS,    
     CROSSJOIN([Transaction Type].LEVELS(0).MEMBERS,   
     CROSSJOIN([Payment Type].LEVELS(0).MEMBERS,   
     CROSSJOIN([Promotions].LEVELS(0).MEMBERS,   
     CROSSJOIN([Age].LEVELS(0).MEMBERS,   
     CROSSJOIN([Income Level].LEVELS(0).MEMBERS,   
     CROSSJOIN([Products].LEVELS(0).MEMBERS,   
     CROSSJOIN([Stores].LEVELS(0).MEMBERS,[Geography].LEVELS(0).MEMBERS)))))))))} ON ROWS  
   FROM ASOSamp.Basic  
 )  





Monday, July 15, 2019

MDX Insert Part Two

In Part One of this series I explained why I think MDX Insert is one of the most important things to happen to Essbase in a while. Now I'm going to show you how to use it. I hope to build from very simple, to very complex over the coming weeks. If you have interesting use cases, send them my way. The more input I get, the more interesting my examples will be.

Keep in mind that you need to be using Essbase in OAC for this to work. It will not work in Essbase 11.1.2.4 or prior.

Documentation

As far as I can tell this is the sum total of documentation on MDX Insert from Oracle:

Tech Reference Spec
MDXINSERTREQUESTTIMEOUT - sets the timeout for an MDX Insert statement
MDXINSERTBUFFERAGGMETHOD - sets whether to add same intersection values or load the last one calculated. Note that it doesn't say if this is an ASO only command so I will assume it isn't until I test it out myself.
Blog One
Blog Two

I'd like to point out that I haven't tried playing around the two essbase.cfg settings above. In OAC it seems you set config settings at an application level and these two settings are not available in the interface (as of 7/14/2019). You can telnet into the OAC server and update the essbase.cfg file. I plan to do that at some point.

Starting Simply

The syntax for MDX Insert is as follows:

INSERT

    source tuple TO target tuple

INTO

    App.Db

FROM

    (

       MDX Select statement goes here

    )


From a high level point of view

1. You are writing an MDX select statement.
2. You're using that to write data to another cell.

Without using an MDX With section, the MDX Insert is only good for copying data from one place to another in a database.

First Example


This example is from the ASOSamp.Basic cube which is available for loading from the Application Gallery in OAC. (If you want to load the sample data you will need to copy that from the gallery into the ASOSamp.Basic file folder then perform a load job -- no load rule is necessary.)

In the MDX script editor in the OAC interface I created the following script.


Here I'm selecting a single cell specified by the following tuple on the rows:
([13118],[056098],[Digital Cameras],[Under 20,000],[20 to 25 Years],[No Promotion],[Cash],[Sale],[Jan],[Original Price])

On the columns I'm selecting [Curr Year]. The column tuple seems to be the source or left side of the equation. I map this into [Prev Year] in the INSERT section.

I can run this query a number of different ways.
1. As a job in the OAC web interface.
2. From Smart View under Manage & Execute Queries. Note that it does not show you the scripts available from ASO cubes. You'd need to copy and paste the MDX Insert statement into the query editor. What's interesting is that if you had created this MDX Insert statement in a BSO cube, Smart View would recognize it as a calc script. So if you were to click the Calculate button, it would show you the classic Essbase calc scripts AND any MDX Insert scripts you had created on the server. I think we need this functionality for ASO.
3. MaxL
4. I assume Rest API (though I haven't tested it)
I don't see a way of using the CLI to run this.

Here's the before:


And the after:


What Does Not Work

From One Into Another

The way the syntax is written, it looks like you should be able to specify different databases for the INTO and SELECT statements. I tried. Sadly it didn't work. I did mention this to someone at Oracle who told me there weren't any immediate plans to add this functionality. It would be pretty cool though. Imagine if you couldn't do that in SQL?

INSERT #MISSING
If I switch the MDX Insert statement above so that my SELECT has [Prev Year] and the INSERT section has "([Prev Year])" TO "([Curr Year])" when Previous Year values, the Curr Year value will not be overwritten. This is the same behavior you would see when using Execute Allocation or Execute Calculation commands with an ASO cube. A load of #MISSING into an ASO cube will not overwrite an existing value. In the past I've used logical clear statements ahead of ASO procedural calcs when I though there might be an issue with overwriting values. For example, if Actuals data needs to overwrite Forecast data.

Under the Covers

I assume what is happening here is that the SELECT statement is generating a dataset. That dataset is then being loaded into the cube as a new slice. When I ran this calc an incremental slice was created. This is what the documentation says on the matter:
During execution of the MDX Insert query, an output buffer is created in memory which accumulates with values until the query is completed.

Appendix

MDX Script for easy copy and paste:
 INSERT "([Curr Year])" TO "([Prev Year])"  
 INTO [ASOSamp].[Basic]  
 FROM (  
   SELECT  
   {[Curr Year]} on COLUMNS,  
   {([13118],[056098],[Digital Cameras],[Under 20,000],[20 to 25 Years],  
    [No Promotion],[Cash],[Sale],[Jan],[Original Price])} on ROWS  
   FROM ASOSamp.Basic  
 )