Showing posts with label MDX Insert. Show all posts
Showing posts with label MDX Insert. Show all posts

Friday, September 13, 2019

MDX Insert Intermission

I have not created any posts about MDX Insert lately as I have hit a snag with my research on the CUSTOMCALCANDALLOCTHRUINSERT Essbase config file setting. This setting, in theory, will take an existing ASO procedural calc and automatically run it as an MDX Insert. This would be great for the customers who have a large number of ASO procedural calcs. I get the impression that once Essbase 19c comes out that this will work. So I'll wait until I have better information before doing a full write-up on the topic.

In the meantime I can report some really great news about MDX Insert and my hope for it becoming the next "big thing" for Essbase.

I was working with a colleague on an ASO procedural calc. He was using 11.1.2.3.500 and ran into the "query exceeds 2^64" error which kills the calc. This was due to the large number of combinations in the POV.

The same calc in 11.1.2.4.031 doesn't give an error but it also never finishes. I think a lot of people were turned off to ASO procedural calcs due to this issue.

The great news is that the same calc that died previously, when translated to MDX Insert, took a grand total of two seconds to complete in OAC. Two frikkin' seconds!

I'm seeking other outlines to try this on. If you've got one I can use for testing, drop me a line.

Also, I'll be at Oracle Open World 19 in San Francisco next week. I hope to see some of you there.

Tuesday, August 6, 2019

MDX Insert Part Three



In Part Two of this series I showed how to perform a simple MDX Insert, copying a single cell of data in an ASO cube. In this post I will show how to create a slightly more complicated calculation.

Keep in mind that you need to be using Essbase in OAC for this to work. It will not work in Essbase 11.1.2.4 or prior.

WITH

I'm going to use the MDX 'With' statement in order to create a calculated member. The With statement comes first in an MDX script. You can use it to define an Essbase member on the fly. You can then use that member in other calculations or in the output of an MDX query. To keep things simple and easy to reproduce for those of you following along at home, I will recreate the Variance member in ASOSamp.Basic.

First I added a member in the outline called Stored Variance.



Next I created my MDX statement. The 'With' statement comes first. In this case I called my member on the fly _Variance. I put an underscore in front of members on the fly so they won't conflict with any Essbase members and so that I can quickly see what's being created dynamically in MDX. Of course, if you happen to use underscores to begin member names in your Essbase outline (heaven help you and) find a different method.

The syntax here is hopefully straightforward. You define the member name AS some formula which goes in single quotes. Here I am subtracting [Prev Year] from [Curr Year].





Next comes the INSERT statement. Remember from part two that the syntax is "(source tuple)" TO "(target tuple)".



Then the SELECT statement is added surrounded by parenthesis. In this case I'm putting the _Variance member on the columns and then the Crossjoin of all the level zero members from the remaining dimensions on the rows. Notice that I don't need to bother leaving out shared or dynamic members.



Once I validate and save the script, I can run the job. To do that I go to the jobs screen in the Jet UI.




Then select New Job -> Run MDX.




Finally, I select the MDX script I want to run.



This calculation took 6.9 seconds to run with the standard ASOSamp.Basic data.


Why?

Why might I want to run a calculation like this? In this case it doesn't make a lot of sense. ASO should be able to handle a simple variance calculation with little effort. But what about a rate times volume calculation? Or a foreign currency calculation? Those get much trickier -- see why here.

The Old Way

You could run this calculation with the Execute Allocation MaxL statement. Here's the statement:
 execute allocation process on database ASOSamp.Basic with  
 pov "CROSSJOIN(Except([Measures].LEVELS(0).MEMBERS,{[Avg Units/Transaction],[% of Total]}),  
     CROSSJOIN(Descendants([MTD],LEVELS([Time],0)),    
     CROSSJOIN([Transaction Type].LEVELS(0).MEMBERS,   
     CROSSJOIN([Payment Type].LEVELS(0).MEMBERS,   
     CROSSJOIN([Promotions].LEVELS(0).MEMBERS,   
     CROSSJOIN([Age].LEVELS(0).MEMBERS,   
     CROSSJOIN([Income Level].LEVELS(0).MEMBERS,   
     CROSSJOIN([Products].LEVELS(0).MEMBERS,   
     CROSSJOIN([Stores].LEVELS(0).MEMBERS,[Geography].LEVELS(0).MEMBERS)))))))))"  
 amount "([Curr Year]-[Prev Year])"  
 target ""  
 range "{[Stored Variance]}"  
 spread;  

Notice that I had to add in an Except() statement to remove members with formulas. I also had to change the way I was pulling the Time dimension members because they too had some formulas. When I ran the script I got these warnings:

 [DBNAME: Basic] The argument [POV] ignores the duplicate member [Flat Panel]  
 [DBNAME: Basic] The argument [POV] ignores the duplicate member [HDTV]  
 [DBNAME: Basic] The argument [POV] ignores the duplicate member [Digital Recorders]  
 [DBNAME: Basic] The argument [POV] ignores the duplicate member [Notebooks]  

So I probably should have removed those shared members from the POV as well. What a pain...

This script took 7.9 seconds to run. I understand that 1 second is not a lot of time but in this case the MDX Insert was about 15% faster than the Execute Allocation. I've always seen performance improvement for MDX Insert when comparing the two methods. Over the course of many longer calculations, the performance improvement can be significant.



NON EMPTY

In case you're wondering, I did try adding a NON EMPTY directive before the rows set and it yielded no performance improvement.
 

Appendix

MDX Script for easy copy and paste:
 WITH  
 MEMBER Years.[_Variance] AS  
   '[Curr Year]-[Prev Year]'  
 INSERT  
   "(Years.[_Variance])" TO "(Years.[Stored Variance])"  
 INTO ASOSamp.Basic FROM   
 (  
   SELECT  
    {Years.[_Variance]} ON COLUMNS,  
    {CROSSJOIN([Measures].LEVELS(0).MEMBERS,  
     CROSSJOIN([Time].LEVELS(0).MEMBERS,    
     CROSSJOIN([Transaction Type].LEVELS(0).MEMBERS,   
     CROSSJOIN([Payment Type].LEVELS(0).MEMBERS,   
     CROSSJOIN([Promotions].LEVELS(0).MEMBERS,   
     CROSSJOIN([Age].LEVELS(0).MEMBERS,   
     CROSSJOIN([Income Level].LEVELS(0).MEMBERS,   
     CROSSJOIN([Products].LEVELS(0).MEMBERS,   
     CROSSJOIN([Stores].LEVELS(0).MEMBERS,[Geography].LEVELS(0).MEMBERS)))))))))} ON ROWS  
   FROM ASOSamp.Basic  
 )  





Monday, July 15, 2019

MDX Insert Part Two

In Part One of this series I explained why I think MDX Insert is one of the most important things to happen to Essbase in a while. Now I'm going to show you how to use it. I hope to build from very simple, to very complex over the coming weeks. If you have interesting use cases, send them my way. The more input I get, the more interesting my examples will be.

Keep in mind that you need to be using Essbase in OAC for this to work. It will not work in Essbase 11.1.2.4 or prior.

Documentation

As far as I can tell this is the sum total of documentation on MDX Insert from Oracle:

Tech Reference Spec
MDXINSERTREQUESTTIMEOUT - sets the timeout for an MDX Insert statement
MDXINSERTBUFFERAGGMETHOD - sets whether to add same intersection values or load the last one calculated. Note that it doesn't say if this is an ASO only command so I will assume it isn't until I test it out myself.
Blog One
Blog Two

I'd like to point out that I haven't tried playing around the two essbase.cfg settings above. In OAC it seems you set config settings at an application level and these two settings are not available in the interface (as of 7/14/2019). You can telnet into the OAC server and update the essbase.cfg file. I plan to do that at some point.

Starting Simply

The syntax for MDX Insert is as follows:

INSERT

    source tuple TO target tuple

INTO

    App.Db

FROM

    (

       MDX Select statement goes here

    )


From a high level point of view

1. You are writing an MDX select statement.
2. You're using that to write data to another cell.

Without using an MDX With section, the MDX Insert is only good for copying data from one place to another in a database.

First Example


This example is from the ASOSamp.Basic cube which is available for loading from the Application Gallery in OAC. (If you want to load the sample data you will need to copy that from the gallery into the ASOSamp.Basic file folder then perform a load job -- no load rule is necessary.)

In the MDX script editor in the OAC interface I created the following script.


Here I'm selecting a single cell specified by the following tuple on the rows:
([13118],[056098],[Digital Cameras],[Under 20,000],[20 to 25 Years],[No Promotion],[Cash],[Sale],[Jan],[Original Price])

On the columns I'm selecting [Curr Year]. The column tuple seems to be the source or left side of the equation. I map this into [Prev Year] in the INSERT section.

I can run this query a number of different ways.
1. As a job in the OAC web interface.
2. From Smart View under Manage & Execute Queries. Note that it does not show you the scripts available from ASO cubes. You'd need to copy and paste the MDX Insert statement into the query editor. What's interesting is that if you had created this MDX Insert statement in a BSO cube, Smart View would recognize it as a calc script. So if you were to click the Calculate button, it would show you the classic Essbase calc scripts AND any MDX Insert scripts you had created on the server. I think we need this functionality for ASO.
3. MaxL
4. I assume Rest API (though I haven't tested it)
I don't see a way of using the CLI to run this.

Here's the before:


And the after:


What Does Not Work

From One Into Another

The way the syntax is written, it looks like you should be able to specify different databases for the INTO and SELECT statements. I tried. Sadly it didn't work. I did mention this to someone at Oracle who told me there weren't any immediate plans to add this functionality. It would be pretty cool though. Imagine if you couldn't do that in SQL?

INSERT #MISSING
If I switch the MDX Insert statement above so that my SELECT has [Prev Year] and the INSERT section has "([Prev Year])" TO "([Curr Year])" when Previous Year values, the Curr Year value will not be overwritten. This is the same behavior you would see when using Execute Allocation or Execute Calculation commands with an ASO cube. A load of #MISSING into an ASO cube will not overwrite an existing value. In the past I've used logical clear statements ahead of ASO procedural calcs when I though there might be an issue with overwriting values. For example, if Actuals data needs to overwrite Forecast data.

Under the Covers

I assume what is happening here is that the SELECT statement is generating a dataset. That dataset is then being loaded into the cube as a new slice. When I ran this calc an incremental slice was created. This is what the documentation says on the matter:
During execution of the MDX Insert query, an output buffer is created in memory which accumulates with values until the query is completed.

Appendix

MDX Script for easy copy and paste:
 INSERT "([Curr Year])" TO "([Prev Year])"  
 INTO [ASOSamp].[Basic]  
 FROM (  
   SELECT  
   {[Curr Year]} on COLUMNS,  
   {([13118],[056098],[Digital Cameras],[Under 20,000],[20 to 25 Years],  
    [No Promotion],[Cash],[Sale],[Jan],[Original Price])} on ROWS  
   FROM ASOSamp.Basic  
 )  

Monday, July 8, 2019

MDX Insert Part One


This is important. Really, really, really important. If you’re using Essbase in the cloud you now have the capability of writing MDX Insert statements. (You'll get this capability on-prem with the 19c release) You can write these against both ASO and BSO cubes. MDX inserts are essentially calc scripts. I’m going to spend some time blogging about them in the future weeks and months. I’ll start out with a brief introduction today.

Let’s start with a history of procedural calculations in ASO.

Initially
When ASO was introduced it wasn't good for much. You were really limited in what you could use it for. Basically loading a large amount of data and adding it up. You could add some formulas but they were slow and couldn't be added outside of certain dimensions. If you wanted a procedural calculation done, you needed to make sure that was done before the data was loaded.

And then
I first started reading about ASO procedural calculations around 2011. I gave them a try in early 2012 and failed miserably. The documentation was difficult to understand and they just didn't seem to work for me. I had heard similar things from other consultants at the time so I gave up.

And then
I started working for a large retailer who had a BSO cube with a 3,000 line calc script for running allocations. The BSO cubes (they were partitioned so there was one for each year) first needed to be aggregated, then the allocations were run, and finally the cubes were aggregated again. Total batch time was about four hours and it was killing us. This was a mission critical cube and it wasn't available until 10AM or later some days. So I started to investigate ASO procedural allocations again. I was encouraged by my manager at the time to try them out on this beast. It took a couple of weeks but I got to the point where I had a proof of concept and worked through most of the technical hurdles -- dealing with dynamic and shared members were among the most challenging. Then it was just a matter of spending a couple of weeks heads down translating the calculations from BSO to ASO. I used "Execute Allocation" and "Execute Calculation" commands. I've blogged about these in the past and have presented at a few Kscope conferences on how to use these.

After the dust settled I had taken a four hour batch process across several BSO cubes and moved it to a single ASO cube that took 10 minutes to run from start to finish. On top of that, the users could get these allocations run during the day in less than two minutes. I've been around Essbase 21 years and I have never seen such performance gains. The users were nonplussed.

I'm disappointed when I hear people who are hesitant to try using ASO. I've worked in this space a long time and I've gotten to the point where there is almost no use case where I'd choose BSO over ASO. ASO is so much more scalable. You just need to learn MDX.

And finally
The MDX Insert statement makes life so much simpler. You can use the same language against an ASO and BSO cube. You don't have to worry about dealing with dynamic and shared members the way you did in Execute Calculation and Execute Allocation scripts. You can run them in Smart View. You can write complex calculations. You can write simple calculations. In my initial performance tests, I saw a 30% improvement compared to Execute Calculation and Execute Allocation. The real test, which I intend to do, is to compare BSO and ASO procedural calc performance.
  
This is the next big step for Essbase. It makes it a more mature and "database-like" technology. Essbase now resides within the database group at Oracle. This is starting to make sense. Moving toward a more standard, less proprietary language is definitely a move in the right direction.

And next
I'm going to cover several examples using MDX in the coming weeks. I'll start off simply and build on that. If you have any specific uses cases you're curious about, send them my way and I'll do my best to tackle them.