Loading CoreData using Python

CoreData is one of the iOS features that I have come to value and use in almost every client project we have undertaken. Combined with the design tools in Xcode it provides a very powerful layer to hide the complexities of database development in a language such as Objective-C. It makes working with complex persistent data structures much easier.

When we started using CoreData in apps it became clear some would benefit from having data preloaded into a CoreData sqlite file. This file could be added to the app bundle and could be copied to the apps Documents folder on first use. However when it came to setting up the project to do this it brought some challenges.

Because the underlying storage is managed by CoreData it raises the question of how do you preload data into the CoreData sqlite store? The internal structure is CoreData’s responsibility. Whilst you can look directly into the sqlite file and deduce its layout to use with some sqlite data loading tools this is far from ideal. Can you be sure you have things correct or that some future release of CoreData won’t change things?

It seems that the sensible approach is to use CoreData itself to write your initial data into the file. This led me to consider the following approaches:

Write some initialization code into the app to read data from some other file and create the CoreData entities from this. The app can then run this code when it is first used to setup the data. However this is not ideal because it could result in a long launch time for the user the first time they run the app. As an example the post code data example below took over a minute to load using this approach on an iPhone 3GS. That is not something a we or an end user would accept.

Write an OS X utility to load the data on your build system. This could then be run periodically to create the sqlite data file for the model. However this means writing a separate ObjC utility and maintaining the code as the model changes through development. Such a utility could be setup as a sub target in the main apps project. It could even share some source code for the model but its still more effort than I’d like. Also I found that Xcode at the time (and it may still be the case) had issues with sub-targets with different platforms. For this approach I found I could use the utility as a tool within the main project but not automate its recreation as needed as part of the main build.

I’ve used python at various points in my career but had not looked closely at the PyObjC bindings available in OS X. However, around this time I was reminded of the technology. PyObjC is a bridging layer that allows access to Cocoa/Foundation frameworks directly from python scripts. So I got thinking, can I drive CoreData from Python? After some experimentation it turns out the answer is yes. Armed with this new approach I was able to write a Python script and run it as part of the iOS build through the command shell build step feature of Xcode. Dependencies could also be setup on the Python script files, the sqlite store file output by the script and any data input files you may want to use. With dependencies correctly configured the script runs at the correct point in the build process and only when it needs to.

In addition Python provides many high level modules for handling csv, xml, various database formats, or you cold even pull data from the apps web service. This is something we have used so the build process automatically include the current data from an associated web service into the target app.

As an example consider we have a post code CSV file with UK post codes and their corresponding GPS coordinates. We want this data loaded into a CoreData store that can be embedded into an app. The Python script in Fig 1 below, importer.py, will do just that. The entity ‘Postcode’ in this example has three attributes ‘postcode’, ‘latitude’ and ‘longitude’. The script takes three input arguments, the path to the object model file, the path to location to write the resulting CoreData store and the post code input csv file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68

#!/usr/bin/python2.6
 
# Import frameworks
from CoreData import *
from Foundation import *
 
import sys,os
 
# MOLoader is a python classed derived from NSManagedObject. It takes an
# entity name, managed object context and dictionary of key/value pairs
# and instantiates an instance of the entity within the context and
# sets the key/value pairs.
class MOLoader(NSManagedObject):
 
    def initWithEntityNamed_inManagedObjectContext_withDictionary_(self, entityName, moc, data):
        entity = NSEntityDescription.entityForName_inManagedObjectContext_(entityName, moc)
        self = super(MOLoader, self).initWithEntity_insertIntoManagedObjectContext_(entity, moc)
        if self is None:
            return None
 
        for key in data.keys():
            self.setValue_forKey_(data[key], key)
 
        return self
 
if __name__ == '__main__':
 
    print 'Using Managed Object Model', sys.argv[1]
    print 'Input data', sys.argv[3]
    print 'Creating CoreData file', sys.argv[2]
 
    # Load model
    bundle = NSBundle.bundleWithPath_(sys.argv[1])
    bundleArray = NSArray.arrayWithObject_(bundle)
 
    mom = NSManagedObjectModel.alloc().initWithContentsOfURL_(NSURL.fileURLWithPath_(sys.argv[1]))
    if mom is None:
        print "Unable to load model"
        sys.exit(1)
 
    # Setup persistent store
    urlForDb = NSURL.fileURLWithPath_(sys.argv[2])
    psc = NSPersistentStoreCoordinator.alloc().initWithManagedObjectModel_(mom)
    ps = psc.addPersistentStoreWithType_configuration_URL_options_error_(NSSQLiteStoreType,None,
        urlForDb, None, None)
 
    if ps[0] is None:
        print ps[1].description()
        sys.exit(1)
 
    moc = NSManagedObjectContext.alloc().init()
    moc.setPersistentStoreCoordinator_(psc)
 
    # Import postcodes
    print 'Adding postcodes'
    for line in open(sys.argv[3]).readlines():
        columns = line.split(',')
        lat = float(columns[1])
        lng = float(columns[2])
        print columns[0],lat,lng
        MOLoader.alloc().initWithEntityNamed_inManagedObjectContext_withDictionary_('Postcode',moc,
            {'postcode':columns[0], 'latitude':lat, 'longitude':lng})
 
    # Save to persistent store
    result = moc.save_(None)
    if not result[0]:
        print result[1].description()
        sys.exit(1)

Fig 1 – importer.py

Now we have a script, its just a case of adding it to the project. Make sure its not included as target member otherwise it will be copied into the app bundle (same applies to any source data files used by the script). To setup the script add a ‘Run Script’ build phase and configure along the lines shown in Fig 2.

Fig 2 – Example script build phase to run python script

I hope this technique is of use and would be interested to hear how others are setting up CoreData files to bundle in their apps. I have found it provides a quick high level means to pre-load data into CoreData from a wide range of input sources. The example is very simple but the concept can be greatly expanded upon for more complex models and data requirements.

Here is a simple Xcode project based on the example discussed.

Example Xcode Project

Finally here are a couple of references to PyObjC.

http://pyobjc.sourceforge.net/
https://developer.apple.com/cocoa/pyobjc.html

Mark Woollard
CTO Tap Tapas LLC