Generating Custom Reports Using XMILE

September 4th, 2013 No comments

XMILE is an open standard for describing system dynamics models in XML.  Version 10 of iThink and STELLA output their models in the XMILE format.  One of the advantages of XML is that it is a text-based format that can be easily queried and manipulated.  This post will show you how to use XMLStarlet, a free XML command line management tool available for Windows, Macintosh, and Linux, to easily extract information from a XMILE model.  It will also demonstrate how to modify the XML style sheet (XSLT) generated by XMLStarlet to create custom HTML reports.

Our goal is to create a report that lists the stocks, flows, and converters in the susceptible-infected-recovered (SIR) model of infection shown below (available by clicking here).  Each model variable will be listed with its own equation and sorted by name.

SIR

XMLStarlet uses the select command (sel) for making queries to an XML file and formatting the results.  We will use all of the following select command options:

-t (template): define a set of rules (below) to be applied to the XML file
-m “XPath query” (match): find and select a set of nodes in the XML file
-s <options> “XPath expression” (sort): sort selected nodes by XPath expression
-v “XPath expression” (value): output value of XPath expression
-o “text” (output): output the quoted text
-n (newline): start a new line in the output

Reporting Stock Names

Let’s start by outputting the names of the stocks in the model.  In a XMILE file, stocks are identified by the <stock> tag, which is nested inside the <xmile> and <model> tags:

<xmile …>
   <model>
      <stock name="Infected">
         <eqn>1</eqn>
      </stock>
   </model>
</xmile>

There is one <stock> tag for every stock in the model and each stock has, at a minimum, both a name (in the “name” attribute) and an initialization equation (in the <eqn> tag).  To get the names of all stocks in the model, we can build a template using these XMLStarlet command options:

sel –t -m “_:xmile/_:model/_:stock” -v “@name” -n

The “sel” chooses the select command and the –t begins the template (the set of rules used to extract and format information from the XML file).  The –n at the end puts each stock name on its own line.

The –m option defines the XML path to any stock from the root.  In this case, the –m option is selecting all the XML nodes named stock (i.e., <stock> tags) that are under any <model> tags in the <xmile> tag.  From the XMILE file, one might expect the XML path to be “xmile/model/stock,” but the tags in the XMILE file are in the XMILE namespace and XPath, which is being used for this query, requires namespaces to be explicitly specified.  Luckily, XMLStarlet, starting in version 1.5.0, allows us to use “_” for the name of the namespace used by the XML file, in this case the XMILE namespace.  Thus, every XMILE name in a query must be preceded by “_:”.

Finally, the –v option allows us to output the name of each node selected with -m (stocks, in this case).  The “@” tells XPath that “name” is an attribute, not a tag, i.e., it is of the form name=”…” rather than <name>…</name>.

To build a full command, we need to add the path to XML Starlet to the beginning and the name of the XML file being queried to the end:

XMLStarlet_path/xml <options above> SIR.stmx

The entire command without the path to XMLStarlet is:

xml sel -t -m “_:xmile/_:model/_:stock” -v “@name” -n SIR.stmx

This command produces the following output:

Infected
Susceptible
Recovered

Sorting the Names

We can sort the nodes after we match them and before we output anything from the selected nodes.  The sort command in XMLStarlet has three options represented using single letters separated by colons (:).  These options, in this order, control whether the sort is ascending or descending, whether we are sorting text or numbers, and whether uppercase letters should come before lowercase (or vice-versa).  Fortunately, the defaults are all what we want (ascending text, uppercase first) so we do not need to specify the specific values.  We still need to specify each option, but can use “-” to indicate we wish to use the defaults.

To the above XMLStarlet options, we need to add the sort command (shown in italics):

sel –t -m “_:xmile/_:model/_:stock” –s –:-:- “@name” -v “@name” –n

This tells XMLStarlet to sort by the name of each stock, yielding this output:

Infected
Recovered
Susceptible

Adding in the Equation

Next, we wish to add the equation after each name.  Since the equation is in its own tag, we can just extract it using –v.  However, we need to output some text between the name and the equation, which we can do with –o:

-o “: ” -v “_:eqn”

The text immediately following the –o option will appear after each name.  Note we have to qualify “eqn” with the namespace (_).  These options belong at the end, right before the -n (shown in italics):

sel -t -m “_:xmile/_:model/_:stock” -s -:-:- “@name” -v “@name” -o “: ” -v “_:eqn” -n

The output is now:

Infected: 1
Recovered: 0
Susceptible: 100

Inserting a Header

We eventually want to show all stocks, flows, and converters, so it is a good idea to identify the above list as stocks.  We can easily do this by adding the header “Stocks:” at the top.  Place it right after the –t option in the command to XMLStarlet:

sel -t -o “Stocks:” -n -m “…” -s -:-:- “@name” -v “@name” -o “: ” -v “_:eqn” -n

The output from these set of options (with the correct XPath after –m) is:

Stocks:
Infected: 1
Recovered: 0
Susceptible: 100

Including Flows and Converters

Flows and converters are identified in the XMILE file with the <flow> and <aux> tags, respectively.  Just like the stock, they each have a name attribute and an <eqn> tag.  Therefore they can be included by repeating the same options we used for the stock, with minor edits.  The entire set of options to the select command (with truncated paths – also note all options must appear on the same command line) is:

-t -o “Stocks:” -n -m “…/_:stock” -s -:-:- “@name” -v “@name” -o “: ” -v “_:eqn” -n
-t -n -o “Flows:” -n -m “…/_:flow” -s -:-:- “@name” -v “@name” -o “: ” -v “_:eqn” -n
-t -n -o “Converters:” -n -m “…/_:aux” -s -:-:- “@name” -v “@name” -o “: ” -v “_:eqn” -n

The extra –n options (start a new line) before flows and converters leaves a blank line between each set.  Using this set of options on SIR.stmx, we get the following output:

Stocks:
Infected: 1
Recovered: 0
Susceptible: 100

Flows:
becoming\ninfected: infection_rate*Infected*Susceptible
entering\ninfected_area: 0
recovering: recovery_rate*Infected

Converters:
infection_rate: 0.005
recovery_rate: 1/4

One Last Whistle

XSLT is very flexible, allowing many more enhancements to our report.  Why don’t we include the total number of entities in the model?

The XPath function “count” returns the number of nodes that match the XPath query.  Thus the following command counts the number of stock nodes in our model file:

count(_:xmile/_:model/_:stock)

We can include the total number of entities in the model at the end of the report by including these options at the end (before the filename and all on the same command line):

-t -n -o “Total Entities: ” -v “count(_:xmile/_:model/_:stock)
+ count(_:xmile/_:model/_:flow) + count(_:xmile/_:model/_:aux)” -n

This adds the following output to the end of our entity list:

Total Entities: 8

The complete XMLStarlet command (without the path to XMLStarlet as it will vary from machine to machine) can be downloaded by clicking here.

Converting to HTML

One of the most powerful features of XML is that we can readily transform the data into an alternate format, as we did above.  We can also add style information to display the data in rich text format in a browser.

The file that contains the rules to transform an XML file is called an XSLT file.  XMLStarlet creates an XSLT file behind the scenes whenever we give it a command.  Using the –C option, we can tell XMLStarlet to output the XSLT file that generates the above output.  We can then edit the XSLT file to add style information.  In particular, we would like the headings to be shown in bold and the variable names to be shown in italic.

Run the same command as before, but add –C directly after the “sel” command (before the first –t).  Redirect the output to a file called SIR.xsl.  The command has this form:

xml sel –C -t … –n SIR.stmx > SIR.xsl

You will also need to make a copy of SIR.stmx named SIR.xml because we need to add a command to the XML file so that it uses the XSLT.  In addition, the browser needs to know it is an XML file, which it does by default for files ending with .xml.

Open SIR.xml in any text editor and insert this line right after the first line (just above the <xmile> tag):

<?xml-stylesheet type=”text/xsl” href=”SIR.xsl”?>

This tells any program that opens the XML file to use the XSLT file named SIR.xsl.  The XML and initial XSLT files are available by clicking here.

If you open the XML file in your browser, you will see something like this:

Stocks: Infected: 1 Recovered: 0 Susceptible: 100 Flows: becoming\ninfected: infection_rate*Infected*Susceptible entering\ninfected_area: 0 recovering: recovery_rate*Infected Converters: infection_rate: 0.005 recovery_rate: 1/4

This is not quite what we had in mind.  The main problem is that XMLStarlet outputs plain text, not HTML.  One way to fix this would be to put everything in a <pre> block, but then we cannot format anything within it.  So we have to edit the XSLT file slightly to make it display properly.

Depending on your browser, you will need to first remove some code from the XSLT file.  Open SIR.xsl in any text editor.  If you are using Firefox, there is an extra dummy namespace in the header line (line 2, starting the xsl:stylesheet tag) that makes Firefox ignore requests to interpret the result as HTML.  Remove this text from that line:

xmlns=”http://www.systemdynamics.org/XMILE”

If you are using Internet Explorer, it gets confused by these lines at the bottom of the file in the template named value-of-template.  Nothing we are doing uses this code, so just remove these lines if you need this to work with Internet Explorer:

<xsl:for-each select="exslt:node-set($select)[position()&gt;1]">
   <xsl:value-of select="'&#10;'"/>
   <xsl:value-of select="."/>
</xsl:for-each>

To get the browser to interpret the results as HTML, add the following attribute to the xsl:output tag (line 3) – just insert it right after xsl:output:

method=”html”

Now we can put the newlines back in, using HTML.  Replace all occurrences of the following with <br/>:

<xsl:value-of select=”‘&#10;’”/>

If you save and open SIR.xml again, you should now see the same output XMLStarlet gave.  To add style information, find the headers “Stocks:”, “Flows:”, “Converters:”, and “Total Entities:”.  These will each be enclosed in an <xsl:text> tag.  Insert <strong> at the beginning of each of these three lines and </strong> at the end.  If you open SIR.xml in your browser now, the headings should all be bold.

Italicizing the variable names is slightly more difficult because the generated XSLT uses a template to output them.  The invocation of this template requires three lines instead of one, so it is harder to pick them out:

<xsl:call-template name="value-of-template">
   <xsl:with-param name="select" select="@name"/>
</xsl:call-template>

The key to finding them is to look for the use of the attribute “@name” inside an xsl:call-template.  Surround all three lines of each of the three instances you find with <em> and </em> and save.  The final XSLT file is available by clicking here.

Congratulations!  If you open SIR.xml again, it should now be formatted as we intended:

Stocks:
Infected: 1
Recovered: 0
Susceptible: 100

Flows:
becoming\ninfected: infection_rate*Infected*Susceptible
entering\ninfected_area: 0
recovering: recovery_rate*Infected

Converters:
infection_rate: 0.005
recovery_rate: 1/4

Total Entities: 8

Advanced:  But What About Those Newlines in Names?

You may have noticed that two of the flow names, rather than using the character “_” between words, use the sequence “\n”.  In XMILE, this is the newline character and appears in a name when the user inserted a line break (by pressing Enter or Return).  This makes it much harder to read the names in the generated report.  It also does not correspond to the name that is used in the equation.  Unfortunately, XMLStarlet does not provide a way to substitute these characters.  In addition, while XSLT 2.0 supports the XPath function replace() to do this, browsers only support XSLT 1.0, which does not include that function.  To replace “\n” in names with “_”, we have to include a template to do the replacement in our XSLT file and then we must change our XSLT code to use that template.

Thankfully, we do not have to write such a template as many are available on the Internet.  The template used for this example is called string-replace-all and comes from here.  Simply copy the entire template (at the top of the post) and paste it at the very bottom of your XSLT file (just above the </xsl:stylesheet> tag).

Since all names and equations are output using the template named value-of-template, the easiest way to affect all names is to modify value-of-template to call string-replace-all.  This will have no effect on equations as they have no newlines in them.  After the changes recommended above for Internet Explorer, value-of-template looks like this:

<xsl:template name="value-of-template">
   <xsl:param name="select"/>
   <xsl:value-of select="$select"/>
</xsl:template>

To replace the newlines, the xsl:value_of line will be changed to call the string-replace-all template to replace “\n” with “_”:

<xsl:template name="value-of-template">
   <xsl:param name="select"/>
   <xsl:call-template name="string-replace-all">
      <xsl:with-param name="text" select="$select"/>
      <xsl:with-param name="replace" select="'\n'"/>
      <xsl:with-param name="by" select="'_'"/>
   </xsl:call-template>
</xsl:template>

The output generated using this new XSLT file (available by clicking here) is:

Stocks:
Infected: 1
Recovered: 0
Susceptible: 100

Flows:
becoming_infected: infection_rate*Infected*Susceptible
entering_infected_area: 0
recovering: recovery_rate*Infected

Converters:
infection_rate: 0.005
recovery_rate: 1/4

Total Entities: 8

Conclusion

XMLStarlet is a useful tool to extract information from a XMILE model file and to generate an initial XSLT file.  To create HTML reports from a XMILE model requires some editing of the generated XSLT file.  What you can accomplish in this way is limited only by your ability to find or write the required XSLT 1.0.  The effort, however, is well worth it; once you create an XSLT file that generates the report you desire, that same XSLT file can be used to generate a report in exactly the same format from any XMILE model.  Try it out by applying the SIR.xsl file created in this post to any other model!

Note:  This post uses the latest version of XMLStarlet, version 1.5.0.  For Windows, it can be downloaded directly from the official XMLStarlet website, http://xmlstar.sourceforge.net/.  For Macintosh, it is available through Fink, a program used to distribute Unix-family open source software for Macintosh (http://fink.sourceforge.net/).  Since XMLStarlet is a command line utility, it must be run from the command prompt in Windows or the Terminal program on Macintosh.

Disclaimer:  This method will not find non-apply-to-all arrays as they use the <array> tag no matter the entity type.

XMILE – An open standard for system dynamics models

July 19th, 2013 No comments

In June, isee systems and IBM sponsored a new technical committee in OASIS, a large standards organization. This committee is developing a new system dynamics modeling standard called XMILE. This blog post will answer some important questions about XMILE.

1. What is XMILE?

XMILE is an open XML protocol for the sharing, interoperability, and reuse of system dynamics models and simulations.

2. What’s the difference between XMILE and SMILE?

XMILE is the XML representation of a system dynamics model. SMILE is the underlying system dynamics language that is represented in XML using XMILE. In this way, it is very similar to the DYNAMO language originally used to create system dynamics models. SMILE could eventually be encoded using something other than XML.

3. How does XMILE benefit iThink and STELLA users?

There are several immediate benefits to iThink and STELLA users:

  • XML files can be reformatted and styled with XSLT files. There are programs available that generate reports directly from XML files.
  • Model files can be examined and edited in a text editor, facilitating searches and simple replaces.
  • Because XMILE is a text file format, proper versioning of model files, showing meaningful differences between revisions, can be done with version control software such as SVN and Git.
  • Because XMILE is textual, platform-neutral, and descriptive, rather than a binary representation of the implementation, it is more resilient to possible file corruption.
  • As the standard becomes more widely adopted additional benefits will include a broader market for models and the ability to share models with colleagues working in different modeling software packages.

4. How will the adoption of the XMILE standard benefit the field of system dynamics?

The benefits of this standard are:

  • System dynamics models can be re-used to show how different policies produce different outcomes in complex environments.
  • Models can be stored in cloud-based libraries, shared within and between organizations, and used to communicate different outcomes with common vocabulary.
  • Model components can be re-used and plugged into other simulations.
  • It will allow the creation of online repositories modeling many common business decisions.
  • It will increase acceptance and use of system dynamics as a discipline.
  • It will help ISVs make new tools that help businesses to develop and understand models and simulations.
  • It will enable vendors to develop standards-based applications for new markets such as mobile and social media.

5. What is the connection to Big Data?

XMILE opens up system dynamics models to a broader audience and for new uses, including embedding models within larger processes. System dynamics models provide a new way to analyze Big Data, especially when pulling live data streams into a running model to determine the impacts of our decisions in real time against future outcomes, to hopefully avoid unintended consequences of our actions. Note, however, that the presumption of Big Data, or the addition of Big Data, does not automatically lead to large, complicated models. You do not have to create giant models just because you have a lot of data. We’re aggregating the data and looking at it in a more homogenous way, so the models can still stay relatively understandable.

6. Can I adapt existing iThink and STELLA models to XMILE?

All of the isee systems products (version 10 and later) already use the XMILE standard in its draft form. As the standard evolves, isee systems products will be updated to meet the changing standard and your models will be translated forward so they remain XMILE-compatible

7. Do you plan to extend XMILE to include discrete event or agent-based simulations?

XMILE focuses on the language of classic system dynamics, rooted in DYNAMO. While we anticipate the language to expand to include both discrete simulation and agent-based modeling, version one of the XMILE specification is restricted to system dynamics modeling.

8. Could you show an example of how XMILE is used in a model?

XMILE is used to describe the model and is the format used for saving it. A model snippet is shown below with the XMILE that completely describes both its simulation and its drawing properties (in the display tag).

image

xmile

9. A big part of system dynamics is graphical, will XMILE include this part of models?

Yes, all graphical information is stored within the display tag, as shown in the earlier example.

10. Why would you want to store visual layout in Xmile? Why not separate structure from layout?

The structure is actually separate from the layout in the XML file. All visual information is embedded within display tags and can be ignored. XMILE defines three separate levels of compliance, with the lowest level being simulation information only (i.e., structure). A model does not need to include display information and any application is free to ignore it.

11. Will XMILE include data from running the model?

XMILE only represents the model structure, so no data is included.

12. Where can I get more information?

The OASIS technical committee for XMILE maintains a public record at https://www.oasis-open.org/committees/xmile/. This page is regularly updated with new information.

The draft standard can be found in these two documents:

http://www.iseesystems.com/community/support/SMILEv4.pdf http://www.iseesystems.com/community/support/XMILEv4.pdf

In addition, isee systems maintains a web page, http://www.iseesystems.com/community/support/XMILE.aspx, that will be updated periodically with new information about XMILE.

Categories: News & Announcements Tags:

Working with Array Equations in Version 10

December 17th, 2012 3 comments

STELLA/iThink version 10 introduces several new array features, including simplified and more powerful Apply-To-All equations that are designed to reduce the need to specify equations for every individual element.

Dimension names are optional

When an equation is written using other array names, the dimension names are not normally needed.  For example, given arrays A, B, and C, each with the dimensions Dim1 and Dim2, A can be set to the sum of B and C with this equation:

B + C

Dimension names are still needed when the dimensions do not match.  For example, to also add in the first 2-dimensional slice of the 3-dimensional array D[Dim1, Dim2, Dim3], the equation becomes:

B + C + D[Dim1, Dim2, 1]

The wildcard * is optional

When an array builtin is used, the * is normally not needed.  For example, to find the sum of the elements of a 2-dimensional array A[Dim1, Dim2] requires this equation:

SUM(A)

If, however, the sum of only the first column of A is desired, the * is still needed:

SUM(A[*, 1])

Simplified array builtins

There are five array builtins:  SIZE, SUM, MEAN, STDDEV, and RANK.  In addition, the MIN and MAX functions have been extended to take either one or two array arguments.  All but RANK can also be applied to queues and conveyors.

SUM, MEAN, and STDDEV all work in a similar way (see examples of SUM above).

Using the MAX function, it is possible to find the maximum value in array A,

MAX(A)

the maximum value in array A, or zero if everything is negative,

MAX(A, 0)

or the maximum across two arrays A and B,

MAX(A, B)

MIN works the same way, but finds the minimum.

The SIZE function requires an array parameter, but within an array, the special name SELF can be used to refer to the array whose equation is being set.  In addition, wildcards can be used to determine the size of any array slice.  In the equation for array A[Dim1, Dim2],

SIZE(SELF)

gives the total number of elements in array A while

SIZE(SELF[*, 1])

gives the size of the first dimension of A, i.e., the number of elements – or rows – in the first column.  Likewise,

SIZE(SELF[1, *])

gives the size of the second dimension of A, i.e., the number of elements – or columns – in the first row.

Since RANK returns the index of the element with the given rank, it can also be used to find the index of the minimum element (using rank 1) or the maximum element (using rank SIZE(array)).  Given array A[Dim1, Dim2], the index of the minimum element in the first row can be found with the equation:

RANK(A[1, *], 1)

However, to find the minimum element in the entire array, use:

RANK(A, 1)

This returns a single index that can be mapped to an array element using the special parentheses subscripting:

A(RANK(A, 1))

will be the value of the minimum element in A, i.e, the same value as MIN(A).  However, if array B has the same dimensions as A (i.e., for this example, B[Dim1, Dim2]), the value of the element in B that corresponds to the minimum element in A is found with:

B(RANK(A, 1))

Accessing elements of queues and conveyors

Use an array subscript to access an element of a queue or conveyor.  The indices start on the outflow side (at 1) and increase toward the inflow side (up to SIZE(queue) or SIZE(conveyor)).  This allows the entire contents of a queue or conveyor to be assigned to an array allowing additional calculations, for example, a weighted average.  Given a conveyor named Lag, a new array weighted_by_time[Slat] can be created with the equation:

(Slat*DT)*Lag[Slat]

Note the subscript is required for the conveyor.  Otherwise, the total value of the conveyor will be used.  Note also that the size of the dimension Slat must be at least large enough to hold all of the conveyor elements (the remaining elements in weighted will be set to zero).  The value of Slat*DT is the amount of time remaining before the material in that slat exits the conveyor.

A converter, average_latency, which is the average time remaining for the contents to exit (a weighted mean), can now be defined with the equation:

SUM(weighted_by_time)/Lag

Transposition

It is sometimes helpful to transpose an array.  To facilitate this, the ' (apostrophe) operator was added.  Given arrays A[Dim1, Dim2, Dim3] and B[Dim3, Dim2, Dim1], the array A can be set equal to B transposed with the following equation:

B '

Note that a space is required between the array name and the apostrophe.  This is equivalent to the following equation that uses dimension names:

B[Dim3, Dim2, Dim1]

This is especially helpful for square matrices or other arrays that use the same dimension name many times.  Given arrays C[Dim, Dim, Dim] and D[Dim, Dim, Dim], the array C can be set equal to D transposed with the following equation, which reverses all the dimensions:

D '

This is equivalent to the following equation that uses the new positional dimension names:

D[@3, @2, @1]

Within a subscript, the @ operator can be followed by an integer that represents the dimension position in the array whose equation is being set.  In the example above, @3 represents the third dimension name of A.  This is particularly useful if straight transposition is not needed and all the dimension names are the same.  For example,

D[@2, @1, @3]

flips the first two dimensions of D (when assigning to A) while leaving the third alone.

Subscript expressions

Subscripts can contain any valid expression.  Given an array A and a variable x, an element at a variable index that is one more than twice x can be accessed with:

A[2*x + 1]

Element labels can also appear within these expressions.

In Apply-To-All arrays, dimension names can be used.  The following equation sets the values in array A[Dim1] to every even-indexed elements in array B[Dim1], filling the second half of A with zeroes:

B[2*Dim1]

Dimension names can also be used outside subscripts.  The following equation slides the elements of B up one position in A, placing 10 in the first element of A (without the IF, the first element would contain 0).

IF Dim1 = 1 THEN 10 ELSE B[Dim1 - 1]

Even if Dim1 is labeled, it must be compared to the numeric index 1 in the IF expression because element labels can only be used within a subscript.  Note that numeric indices are always valid for any array dimension, even if it is labeled.

Array Ranges

A range of an array can be specified using the range operator : (colon), which takes a lower bound on the left and an upper bound on the right (e.g., 1:10 means “from 1 to 10”).  Just as wildcards allow control over which dimensions to include, ranges control which range of elements to include in each dimension.  For example, the follow equation sums the top-left 3×4 rectangle of array A[Dim1, Dim2]:

SUM(A[1:3, 1:4))

We hope you find these new array capabilities useful in your modeling work and welcome any comments and suggestions.

2011 Barry Richmond Scholarship Award

August 10th, 2011 2 comments
Sarah accepts award

Sarah Boyar accepts Scholarship Award from Joanne Egner

The Barry Richmond Scholarship Award was established in 2007 by isee systems to honor and continue the legacy of its founder, Barry Richmond.  Barry was devoted to helping others become better systems citizens.  Systems citizens are members of a global community that strive to understand the complexities of today’s world and have the informed capacity to make a positive difference.  It was Barry’s mission to make systems thinking and system dynamics accessible to people of all ages, and in all fields.  The award is presented annually at the System Dynamics Society Conference to an individual whose work demonstrates a desire to expand the field of systems thinking or to apply it to current social issues.

Through most of his career, Barry focused on education as the key to spreading systems thinking.  As a teacher and a mentor he dedicated much of his time to developing tools and methodologies for teaching systems thinking.  With this in mind, it was a great pleasure to present this year’s award to Sarah Boyar, a recent graduate of the Masters Program in System Dynamics at Worcester Polytechnic Institute (WPI).

Sarah Boyar and Karim Chichakly enjoy the conference banquet

Sarah Boyar and Karim Chichakly enjoy the conference banquet

Sarah presented a portfolio of her work to the scholarship committee.  In particular, an essay about her teaching philosophy resonated with us.  Sarah wrote this piece while taking a seminar in college teaching in order to fulfill her Real World Dynamics course requirement at WPI.  Since she already had plenty of experience as a consultant applying system dynamics to real world situations, Sarah managed to convince the WPI powers-that-be that an essential real world manifestation of system dynamics is the way that it is taught.  This is something Barry would have encouraged and been excited about.

Her essay titled Beliefs About Teaching and Learning begins as follows:

I teach System Dynamics. While I want my students to have some knowledge of system dynamics, most of all I want them to be excited and stimulated by it. I also want them to find it beautiful: I want to teach in such a way that my students find some aspect of beauty in the work, whether it’s through the visual arcs in the model interface, or the precision of algebra in the way we write statements, or the way that system dynamics can ameliorate a social ill that concerns them. I want my students to somehow feel a sense of peace and beauty derived from some aspect of the knowledge I am teaching.

Among Sarah’s aspirations is the desire to teach system dynamics to professionals in other fields, namely lawyers (potential judges) and medical doctors.  Incorporating a systems perspective within both the judicial system and in healthcare could certainly make a positive difference for us all.  Good luck and congratulations Sarah!

Connecting iThink and STELLA to a Database

April 28th, 2011 5 comments

A question we periodically get from our customers is: Can iThink or STELLA connect to a database? Saving and pulling information to/from databases presents a lot of advantages for storing, organizing and sharing model data. Thanks to iThink and STELLA’s ability to import and export data via commonly used spreadsheet file formats, it is possible to use comma separated value (CSV) files as a means to create a connection to database applications.

Essentially, data can be moved between a database and iThink/STELLA by using a CSV file as a bridge. CSV files are a widely supported file standard for storing table data, and both iThink/STELLA and many database programs are able to read and write to them.

Process overview

The process of connecting to a database using CSV files as an intermediary

The process can be automated when you use iThink/STELLA’s ability to run models automatically from the command line (Windows only). Most database applications also have command line interfaces, allowing you to create a single macro script that moves data between your model and a database in a single process.

In this post I will use a simple example to demonstrate how to import data from a Microsoft SQL Server database into an iThink model on Windows. The model and all files associated with the import process are available by clicking here. If you don’t have access to Microsoft SQL Server, you can download a free developer’s version called SQL Server Express from the Microsoft web site.

The Model

The model used in this example is a variation of the Beer Game model. The structure shown below represents the ordering process for a simple retailer supply chain.

Retail Supply Chain Model

The model has been set up to import the initial values for On Order with Wholesaler and Unfilled Orders stocks, target inventory and actual customer orders (a graphical function with 21 weeks of data). The source of the imported data is the file named import.csv in the example files.

To set up this example, I manually created the CSV file using the initial model parameters. (Later in this post, you’ll see that this file will be automatically created by the database.) The model has been initialized in a steady state with actual customer orders at a constant level of 4 cases per week over the 21 week period.

Creating the SQL Server Database

Since the model is set up to import data from the import.csv file, we are ready to populate that same CSV file with data from the SQL Server database. But, first we need to create the database and the associated data table.

Using SQL Server Management Studio, create a new database named Simulation.

New Database Dialogue in SQL Server

Dialog in SQL Server when adding new "Simulation" database

Once the database has been created, you can run the full CreateTable.sql script to create a table that will store the data. To open the script, select Open -> File from the File menu and navigate to the CreateTables.sql file. The script is set up to create a table named Import_Data with columns for each variable that is imported into the model. The script will also populate the table with data. It is important to note that the first row of the table contains the variable names exactly as they appear in the retailer supply chain model. Below is a sampling of the commands contained in the script:

CREATE TABLE [dbo].[Import_Data](
[RowID] [int] IDENTITY(1,1) NOT NULL,
[OnOrder] [nvarchar](50) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
[UnfilledOrders] [nvarchar](50) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
[TargetInventory] [nvarchar](50) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
[ActualOrders] [nvarchar](50) COLLATE SQL_Latin1_General_CP1_CI_AS NULL
) ON [PRIMARY]

INSERT INTO Import_Data
(OnOrder,UnfilledOrders,TargetInventory,ActualOrders)
values
(‘On Order with Wholesaler’,'Unfilled Orders’,'target inventory’,'actual customer orders’)
INSERT INTO Import_Data
(OnOrder,UnfilledOrders,TargetInventory,ActualOrders)
values
(8,4,20,4)
INSERT INTO Import_Data
(OnOrder,UnfilledOrders,TargetInventory,ActualOrders)
values
(”,”,”,4)
INSERT INTO Import_Data
(OnOrder,UnfilledOrders,TargetInventory,ActualOrders)
values
(”,”,”,10)

. . .

To run the script, select Execute from the Query menu. Note that the data shows an increase in actual customer orders in week 2 from 4 to 10 cases. In the model, actual customer orders is a graphical function which is reflecting order data over time.

Results tab after running CreateTable.sql script

Results tab after running CreateTable.sql script

Linking the Database to the Model

After the database has been prepared, the Transfer_Data.bat Windows batch file can be used to link the database to the model. Transfer_Data.bat is a script that pulls data from the database into the CSV file, opens iThink, imports the CSV data and runs the model. The batch file takes advantage of the utility Microsoft provides with SQL Server called Bulk Copy Program (BCP) and iThink’s ability to run models from the command line. You’ll need to make a few edits to the batch file shown below before you can run it.

BCP Simulation.dbo.Import_Data out import.csv -c -t, -U <Login> -P <Password> -S <Server name>

“%PROGRAMFILES%\isee systems\iThink 9.1.4\iThink.exe” -i -r -nq Supply_Chain.itm

Using a text editor such as WordPad or NotePad, replace the placeholders for <Login>, <Password> and <Server name> in the first line to match the authentication for the Simulation database that was previously created. If your SQL Server database is on your local machine, you can delete the –U, -P and –S settings altogether. This command will make use of the BCP utility to copy the contents of the Import_Data table to the import.csv file. Depending on the version of iThink or STELLA you are running (version 9.1.2 or later is required) and its location, you may also need to edit the second line of the batch file.

After the Transfer_Data.bat file has been edited and saved, double-click on the file to run it. Once the data has been retrieved from the database, iThink will import the data via the CSV file, and run the simulation. The results will appear in the graph in the model as shown below:

Retail inventory graph

Updating the Data

In a real-world situation, the database that stores information we need to run a model is usually connected to another application that updates or changes the data. In the absence of a real-world application, I created a script that allows you to update the Simulation database and experiment with your own set of data.

In SQL Server Management Studio, open the script called UpdateTable.sql. Edit the values for each variable as desired.

USE [Simulation]
UPDATE Import_Data SET OnOrder=8, UnfilledOrders=4, TargetInventory=20, ActualOrders = 4
WHERE RowID=2
UPDATE Import_Data SET ActualOrders = 4
WHERE RowID=3
UPDATE Import_Data SET ActualOrders = 4
WHERE RowID=4

. . .
When you have finished editing the UpdateTable.sql script, execute the script to update the Import_Data table. Now you are ready to re-run the Transfer_Data.bat batch file and link the updated data with the Supply_Chain.itm model.

Summary

This example should give you a good idea of how the process of importing data from a database into an iThink or STELLA model can be set up and run automatically. Other database applications such as Oracle, MySQL and SQLite could also be used in a similar manner. I have not yet tried to set up examples, but these other databases also have command line interfaces for automating the transfer of data to CSV files that iThink and STELLA can connect to.

Stay tuned for part two of this post where I’ll provide an example of exporting model results to a SQL Server database.

What is the difference between STELLA and iThink?

March 9th, 2011 3 comments

The question we get asked most frequently by just about anyone who wants to know more about our modeling software is “What is the difference between STELLA and iThink?”  From a functional perspective, there are no differences between the STELLA and iThink software — they are two different brands of the same product.

The STELLA brand is targeted toward individuals in educational and research settings.  Supporting materials such as An Introduction to Systems Thinking with STELLA and sample models cover the natural and social sciences.

iThink, on the other hand, is targeted toward an audience of users in business settings.  An Introduction to Systems Thinking with iThink is written with the business user in mind and model examples apply the software to areas such as operations research, resource planning, and financial analysis.

Aside from the different program icons and other graphic design elements that go along with branding, there are just a few minor differences in the default settings for STELLA and iThink.  These differences are intended to pre-configure the software for the model author.  They do not limit you in any way from configuring the default setup to match your own individual preferences.

Below is a list of all the differences between the default settings for STELLA and iThink.

Opening Models

When opening a model with STELLA on Windows, by default, the software looks for files with a .STM extension.  Similarly, iThink looks for files with an .ITM extension.  If you want to open an iThink model using STELLA or vice-versa, you need to change the file type in the Open File dialog as shown below.

STELLA file open dialog

On Macs, the open dialog will show both iThink and STELLA models as valid files to open.

If you open a model with a file type associated with the different product than the one you are using, you’ll get a message similar to the one below warning you that the model will be opened as “Untitled”.  Simply click OK to continue.

STELLA file conversion dialog

Saving Models

When saving a model in STELLA, by default, the software saves the model with a .STM file extension.  Similarly, iThink saves model s with an .ITM extension.  If you’re using STELLA and want to save your model as an iThink file or vice-versa, use the Save As… menu option and select the appropriate type as shown below.

STELLA save as dialog

STELLA on Windows save dialog

 

STELLA on Mac save dialog

STELLA on Mac save dialog

Run Specs

Since iThink is targeted toward business users who tend to measure performance monthly, the default Unit of time for iThink is set to Months.  It’s also easier to think about simulations starting in month 1 (rather than month zero) so we set the default simulation length in iThink to run from 1 to 13.  STELLA on the other hand, reports the Unit of time as “Time” and, by default, runs simulations from 0 to 12.

Run Spec comparison

Run Spec Default Settings Comparison

Table Reporting

In a business context, financial results are generally reported at the end of a time period and the values are summed over the report interval.  For example, in a report showing 2010 revenues we would assume the values reflect total revenues at the end of the year.  In line with this assumption, the default Table settings in iThink include reporting Ending balances, Summed flow values, and a report interval of one time step.

In a research setting, scientists tend to prefer reporting precise values at a particular time.   For this reason, the default Table settings in STELLA are configured to report Beginning balances, Instantaneous flow values, and a report interval of Every DT.

table default settings comparison

Table Default Settings Comparison

STELLA or iThink

When choosing between STELLA or iThink, try to think about the kinds of models you intend to build and the problems you are looking to solve.  If your objective is to drive business improvement, chances are iThink will be a better fit.  If your purpose is to understand the dynamics of a natural environment or social system, STELLA will likely be your brand of choice.  Whatever you decide, both products will provide you with the exact same functionality and can easily be configured to suit your own preferences.

Using PEST to Calibrate Models

January 14th, 2011 21 comments

There are times when it is helpful to calibrate, or fit, your model to historical data. This capability is not built into the iThink/STELLA program, but it is possible to interface to external programs to accomplish this task. One generally available program to calibrate models is PEST, available freely from www.pesthomepage.org. In this blog post, I will demonstrate how to calibrate a simple STELLA model using PEST on Windows. Note that this method relies on the Windows command line interface added in version 9.1.2 and will not work on the Macintosh. The export to comma-separated value (CSV) file feature, added in version 9.1.2, is also used.

The model and all files associated with its calibration are available by clicking here.

The Model

The model being used is the simple SIR model first presented in my blog post Limits to Growth. The model is shown again below. There are two parameters: infection rate and recovery rate. Technically, the initial value for the Susceptible stock is also a parameter. However, since this is a conserved system, we can make an excellent guess as to its value and do not need to calibrate it.

image

The Data Set

We will calibrate this model to two data sets. The first is the number of weekly deaths caused by the Hong Kong flu in New York City over the winter of 1968-1969 (below).

clip_image004

The second is the number of weekly deaths per thousand people in the UK due to the Spanish flu (H1N1) in the winter of 1918-1919 (shown later).

In both cases, I am using the number of deaths as a proxy for the number of people infected, which we do not know. This is reasonable because the number of deaths is directly proportional to the number of infected individuals. If we knew the constant of proportionality, we could multiply the deaths by this constant to get the number of people infected.

Read more…

Shifting the Burden

December 22nd, 2010 3 comments

The Shifting the Burden Systems Archetype shows how attacking symptoms, rather than identifying and fixing fundamental problems, can lead to a further dependence on symptomatic solutions.  This Systems Archetype was formally identified in Appendix 2 of The Fifth Discipline by Peter Senge (1990).  The Causal Loop Diagram (CLD) is shown below.

image

When a problem symptom appears, two options present themselves:  1) apply a short-term fix to the symptom, or 2) identify and apply a longer-term fix to the fundamental issue.  The second option is less attractive because it involves a greater time delay and probably additional cost before the problem symptom is relieved.  However, applying a short-term fix, as a result of relieving the problem symptoms sooner, reduces the desire to identify and apply a more permanent fix.  Often the short-term fix also induces a secondary unintended side-effect that further undermines any efforts to apply a long-term fix.  Note that the short-term fix only relieves the symptoms, it does not fix the problem.  Thus, the symptoms will eventually re-appear and have to be addressed again.

Classic examples of shifting the burden include:

  • Making up lost time for homework by not sleeping (and then controlling lack of sleep with stimulants)
  • Borrowing money to cover uncontrolled spending
  • Feeling better through the use of drugs (dependency is the unintended side-effect)
  • Taking pain relievers to address chronic pain rather than visiting your doctor to try to address the underlying problem
  • Improving current sales by focusing on selling more product to existing customers rather than expanding the customer base
  • Improving current sales by cannibalizing future sales through deep discounts
  • Firefighting to solve business problems, e.g., slapping a low-quality – and untested – fix onto a product and shipping it out the door to placate a customer
  • Repeatedly fixing new problems yourself rather than properly training your staff to fix the problems – this is a special form known as “shifting the burden to the intervener” where you are the intervener who is inadvertently eroding the capabilities and confidence of your staff (the unintended side-effect)
  • Outsourcing core business competencies rather than building internal capacity (also shifting the burden to the intervener, in this case, to the outsource provider)
  • Implementing government programs that increase the recipient’s dependency on the government, e.g., welfare programs that do not attempt to simultaneously address low unemployment or low wages (also shifting the burden to the intervener, in this case, to the government)

Read more…