Moving data between different environments is always more difficult than we hope. Here I list some approaches and actually try them out on a small dataset. We hit some bugs along the way and also a few conceptual stumbling blocks (even for this stylized example). We had some issues with Access as well as GAMS Connect.
This question came up in an actual project. My idea was: "Let me show you how this can be done". I am afraid, I got carried away a bit. But it demonstrates that we should not underestimate these, at first sight, menial tasks. When the data set becomes larger, the problems compound. We can't eyeball the data, and statistically, it is more likely we encounter some problems.
I also did a quick test on some larger data sets. GAMS Connect is extremely slow when inserting lots of data. It is much better to use a CSV file and import that into Access. That can be 50 times as fast as using GAMS Connect.
Word Document
Powerpoint Presentation
- When writing to a database, standard inserts are very expensive. For larger data sets we need to use specialized BULK INSERT methods.
- When reading, it can be very beneficial to support wide tables. Exporting a wide table from a database is much more efficient than using the corresponding long table.