Hi,
I'm using a Mac and I am trying to extract certain data points for large XML files created by and available from the OpenStreetMap project. I have downloaded an XML file that describes Scotland. This file size is 3.2 Gbytes which all my text processors choke on. However the excellent "HexFiend" reads it without problem.
To date I have not used the XML tools that Livecode provides as I'm not a fan of xml, so I am following one of the tutorials. Basically this loads the data into a variable and then this variable is passed to the XML routines to create a tree. Later I plan to convert the tree to an array but I'm digressing.
My mac has 16Gbytes of memory so the 3.2 Gbytes is well within memory limits and so far the simple code is using just over 10Gbytes of memory.
Other larger regions of the world obviously require larger files. I wonder how Livecode handles instructions that cause the available memory to overflow. I know I can and should try such a file but even the 3.2Gbyte file is taking ages to process and I'm not sure I have the patience to wait for Livecode to attempt to process a 50 Gbyte file. So does anyone know if Livecode automatically starts to use the hard drive to store data in its variables or is it up to me?
I should say that I have no intention of having my final version process whole files in one go as it is obvious that reading in bite size chunks of data is a superior method of both developing the app and reading the data. However, I am interested how Livecode manages memory overflow. My guess is that it will use resources until the OS starts to post warnings then die.
Allocation of memory when processing large files
Anything beyond the basics in using the LiveCode language. Share your handlers, functions and magic here.
Moderators: FourthWorld, heatherlaine, Klaus, kevinmiller, robinmiller
-
- Posts: 854
- Joined: Wed Nov 04, 2009 11:41 am
- Location: Gunthorpe, North Lincs, UK
Allocation of memory when processing large files
Post by Simon Knight » Sat Oct 03, 2020 8:35 am
best wishes
Skids
Skids
Jump to
- LiveCode Community
- ↳ Announcements
- ↳ Developer Services Marketplace
- ↳ User Groups and Events
- ↳ SoCal LiveCode Group
- ↳ Greater Toronto Area LiveCode Users Group
- ↳ Off-Topic
- ↳ Made With LiveCode
- Getting Started With LiveCode
- ↳ Getting Started with LiveCode - Complete Beginners
- ↳ Getting Started with LiveCode - Experienced Developers
- ↳ Converting to LiveCode
- ↳ App Building Course
- ↳ Simple Messages
- ↳ Calculator
- ↳ Notes
- ↳ Clock
- ↳ Web Browser
- ↳ Weather
- ↳ Compass
- ↳ Photos
- Developing With LiveCode
- ↳ Talking LiveCode
- ↳ Databases
- ↳ Games
- ↳ Internet
- ↳ Multimedia
- ↳ Software Engineering
- ↳ Version Control
- ↳ Feature Proposals
- ↳ Bug Triage
- ↳ Legacy Project: MetaCard
- Extending LiveCode
- ↳ Using Plugins for Customizing the LiveCode IDE
- ↳ Using Externals
- ↳ Making IDE Plugins
- ↳ Building Externals
- ↳ LiveCode Builder
- Platform Specific
- ↳ Windows
- ↳ Mac OS
- ↳ Linux
- ↳ iOS Deployment
- ↳ Android Deployment
- ↳ CGIs and the Server
- ↳ HTML5
- ↳ Raspberry Pi
- Academic Endeavors
- ↳ Teaching with LiveCode
- ↳ Research and Post Secondary
- ↳ LiveCode University
- ↳ Educational Outreach
- Livecode Open Source Archived Forum- closed to posting
- ↳ Engine Contributors
- ↳ IDE Contributors
- ↳ Documentation and Tutorials
- ↳ Brainstorms
- ↳ Community Projects
- ↳ rIDE
- Third Party Extension Support
- ↳ Marketing Your Products
- ↳ Animation Engine
- ↳ chartsEngine
- ↳ MobGUI
- ↳ lcTaskList
- ↳ SQL Yoga
- ↳ MergEXT
- ↳ Data Grid Helper
- ↳ idea2app and Coding School
- ↳ Admin Archive
- LiveCode Hosting
- Forum Archives
- ↳ Archived: Third Party Extension Support
- ↳ rTree
- ↳ rGrid
- ↳ Data Tree
- ↳ Archived: LiveCode Open Source
- ↳ LiveCode Global Jam
- ↳ Testing Support
- ↳ Archived: Academic Endeavors
- ↳ September Academy
- ↳ Summer School 2013
- ↳ Archived: Project Specific
- ↳ Business and Kiosks
- ↳ Digital Art and Portfolios