IMS Enterprise 1.1 Vista XML Integration, Blackboard Learn 9, and Processing Enrollment Changes

Having just worked through a migration from Vista 8 to Blackboard's Learn 9.1  LMS platform, there were some challenges around translating the formerly "live" integration achieved from Banner to to Vista via the LMB to the new XML Feed file integration provided by Blackboard's Learn 9.1 platform.

I should start this post with the caveat that LMS Administration is a fairly new pursuit to me, by virtue of some staffing transitions at Marshall University - so it's possible (likely even) that if you've been doing LMS administration for some time, much of what I describe as I walk through how we achieved our integration will be second nature to you.   It's also likely that you'll laugh at a few of my silly mistakes - but I'll discuss each of them anyway, in the hope that someone else in my shoes can benefit from the experience.


Our Environment


In the interest of background to help you determine if this post is right for you - as well as in an effort to hit on some of the keywords most likely to bring the right information seekers to this post, I'll talk through our previous and current environments.

The old environment


Blackboard/WebCT Vista 8 Enterprise
Banner SIS sending person and membership record updates via the Luminus Message Broker

The new environment


Blackboard Learn 9.1 SP8
Banner SIS providing XML exports of person and membership records
Export files posted and processed by Learn 9.1 via the IMS Enterprise 1.1 Vista XML Feed File integration



Our Situation


The previous semester, we had run a dual environment where Learn 9.1 was our "pilot" LMS with a limited number of live course offerings, while Vista remained our core LMS.

We had always planned to migrate all of our live courses to Learn 9.1 for the new spring semester, and
with staff transitions happening, and the reality that someone with little knowledge or history of the LMS was going to have to quickly get up to speed, it made more sense to push forward on one system than to try to put on the brakes on our planned rollout and continue to support both systems.

Pushing forward meant quickly figuring out how to make integration between the SIS and the new Learn 9.1 system as seamless as possible with the loss of the "live" LMB integration.   It also meant that there would be a lot of trial and error, and learning from silly mistakes along the way.


Making Integration Work

Let's walk through the steps we took to get our integration working.   Along the way, where we encountered problems, made mistakes, or had to work around an issue, I'll call that out along with any additional information that might prove helpful to someone else in a similar situation.

Step One - Setting up a semester data source


The first step I took was to set up a new datasource in the Learn 9.1 admin to handle the new semester course and enrollment data.   By assigning a new data source key to each term, data can be manipulated in sets that correspond to logical term dates.   This way, if a change needs to occur that impacts courses, courses can be manipulated by term without affecting other data in the database.   Additionally, setting up a new data source key for each term eases reporting later, when you want to pull out information about a specific time period.

To create a new data source, log in as a system administrator, and click on the Data Integration building block link:

Next, click the "Data Sources" Link:

Click the button labeled "Create Data Source" at the top of the page:

From here, you'll be presented with a "Data Source Properties" dialog.    Give your data source a key that makes sense to you, and a similarly descriptive bit of text in the description box.   Since the data source I was creating was for the "2nd" term of our 2012 academic year, the key I chose was 201202.

Step Two - Creating an SIS Integration for your new data source

Once you have your new data source created, you need to either tie an existing integration to the new data source, or else create a new integration for the newly created data source to feed it data.   For simplicities sake, I created a new SIS Integration to match the newly created data source.  To do this, follow the same steps as you would when creating a data source - but this time rather than clicking on the "Data Sources" link on the Data Integration Page, click on the "Student Information Systems Integrations" link.

On this page, clicking the "Create Integration" button at the top of the page will expand a drop down menu showing you your available integration choices.

Since we are using banner as our SIS, and we know that it's producing IMS Enterprise 1.1 - Vista compliant XML files when we run our user and enrollment queries, we selected "IMS Enterprise 1.1 - Vista" when creating the new integration.

Important note:  You can have more than one integration tied to a given data source.   In fact, there are some cases where you may need to perform an administrative level action on data in your database that will require you to create a flat file integration.   I ran in to two of these cases myself when I needed to change the data source an existing user record was associated with.    The important thing to understand is that it's not a big deal to have more than one integration set up for a given data source at the same time.


Once you click on the type of integration you want to create, you are taken to the configuration page for that type of integration.   The information that you need is going to vary depending on the type of integration that you're setting up.    Since, in our case, we were focusing on IMS Enterprise 1.1 - Vista, that's what the screenshots I'm providing will be of.

This type of integration will present you with three primary configuration sections:  Integration Properties, Data Support, and Advanced Configuration.   Let's break each of these sections down and talk about key components.

Integration Properties

Because I quickly understood that each semester I was going to want a new data source, I initially thought that I would need to do the same thing with the integrations I was creating.    So, naturally, I named my integration 201202 just like my data source.    As I later understood, you can take an integration and tie it to another data source at any time, so what you call your integration doesn't need to reflect your database naming structure.   If I were doing it again from scratch, I'd probably just give the integration a name reflective of the type of integration being created "Banner-IMS-Vista" for example.

The "Shared Username" you see here is a unique identifier tied to this particular integration that you'll use as a username if you want to take advantage of the web services end points provided when you create an integration to auto-post your XML files.

The "Shared Password" field is where you can supply your own unique password to be used when calling these integration end points as web services.    

"Integration Status" is self explanatory - as is "Log Verbosity".   

Data Support

In the data support section, choose the radio button labeled "Use the same Learn Data Source for all new inbound data" and choose the data source you created in the previous step from the drop down box provided.

You should be able to leave all the other toggles at their defaults.   I was, but your mileage may vary, so be sure to test with your data before pushing live.

Advanced Configuration

The "Advanced Configuration" section is where you'll define how this integration handles the processing of the XML that you'll be importing.   What you set these to will depend on your particular institutional business rules - but in general, it is better to use "Disable" than "Purge" in your deletes column if there's any doubt as to whether you'll want to keep the data around.

Once you're happy with the configuration you've set up for each type of data you'll be sending over, click the "Submit" button at the bottom of the page to create your new integration.

Step Three - Test your newly created data source and integration

After you've set everything up, you're going to want to test your newly created integration by grabbing some person or membership records from your SIS in XML format.   Until you're sure things are working the way you want, you might want to just create some sample records from a subset of your data.    It's also helpful to remember that the integration is going to process an enrollment, generally, in this order:

1. Load a course

2. Load a section

3. Load a faculty member person record

4. Load a faculty member enrollment record to associate them as instructor for that section

5. Load a student person record

6. Load a student enrollment record enrolling them as a student in a given course

Remembering this general flow is helpful when deciding how to load your data tests so you don't end up getting records when the integration tries to enroll a student in a course that doesn't already exist, for example.

Test By Uploading a Feed File

To test this type of integration, again click on "Student Information Systems Integrations" in the Data Integrations Building blog section of the admin.    Find the integration that you created, and click on the drop down link to the right of the integration name.

The two most useful links in this menu during your testing are "Upload Feed File" where you'll actually upload the XML files that you want this integration to process, and "View Log", where you'll go to check and see if the processing worked as you expected.  

After you've uploaded your feed file, come back and click the "View Log" links to see if there's anything unexpected happening.   If there is, you should see errors appear that are reasonably easy to troubleshoot (usually improperly formatted XML).     Assuming no errors, go ahead and take the next step of verifying that the records are showing up in the admin UI after processing by doing some searches for users, courses or enrollments.

Step Four - Automating your Feed File Uploads

Assuming that everything is working as expected, you'll want a way to automate the upload process so that you won't need to have someone visit the site each day and manually upload these files.   This is where the Shared Username, and Shared Password that were configurable when creating your integration come in to play.

To post XML feed files to the integration you've created automatically, you need three pieces of information.   You'll need a username and password (which we've already covered), and you'll need to know the endpoint URL that you want to post to.    The endpoint URL is provided in that same drop down menu used to upload the feed files.  

Grab that endpoint URL, and set up a curl, python or similar script to post the contents of your feed XML to the endpoint using your shared username and password for that integration.

In our case, we used a curl post that was set up to run at periodic intervals throughout the day.   The duration you'll need to wait between posts is dependent on how often you need your data to refresh.   Since we have username and password data for our students included in the feed posts, and since students can and do change those pins periodically throughout any given day, we chose to refresh our data on a one hour cycle.

Handling Unexpected Issues with Integration

There were a couple of places where a lot of time was spent banging my head against my desk because things were happening (or more often were not happening) as I expected once the automated feed file posting was set up.    Let me step through each of these.

Problem #1 - Many users calling our service desk unable to login

After the integration was set up and working, and had been tested as much as I could test it prior to going live, I felt confident that all systems were go and things should work as expected.   When users started getting locked out and pin/password resets weren't having any effect, I had to investigate why.

As it turns out, in addition to the initial integration configuration screen that you see when first setting up the integration, there is a bit more detailed configuration screen available that allows you access to field mappings so that you can match data fields in your SIS to the fields that Learn 9.1 is expecting and understands.   

To get to this advanced configuration, you again click on the "Student Information Systems Integrations" link in the admin, click the drop down link next to the integration in question, but this time click on "Advanced Configuration" in the third portion of the drop down menu.

Doing that will bring you to this screen, where you can further drill down in to each specific object type.

Since in the case of user pin resets not working we're dealing with user objects, I can click the drop down next to "Users" to access the field mappings configuration for this object type.

On the field mapping screen, the first thing I noted was that the check box under "Change on Update?" was not checked for User Passwords.   Bingo - that must be it.   I happily checked that, and went on my way, thinking that passwords would now sync.

Sadly, this didn't correct the problem.   Because our passwords are encrypted, I need to come back to this screen and also check the "Password Encryption Type" change on update checkbox before password resets began syncing up.

Problem #2 - Processing Drops, Changes in Enrollment Availability

The next major issue was handling users who would drop a course.   The documentation provided around integration leads you to believe that if you enroll a user in a course, and then later rerun membership data where that user isn't tied to that course, the enrollment record for that user will be disabled.

This isn't how it works.   In fact, users could drop a course, and have no further reflection fo their user being a member in a course in future XML feed pushes, but this had no effect on their membership in the course in Learn.   

As it turns out, what we're dealing with is another field mapping issue, this time around the "Enrollment" object, and the "Row Status" change on update setting.   Row Status is what controls whether a particular user record is set to available or unavailable for a given course.

This value *should* key off of the <status></status> value in the membership XML.   According to documentation, if the status is set to 1 in the XML, the user is set to "available" for the course in question. Conversely, if the status value is set to 0 in the XML, the user is supposed to be set to "unavailable".    

In our case, setting the status value to 0 (which is what happens in Banner when a user drops a course) wasn't having any impact on the user availability in Learn.

In order to make this work, you actually need to modify the "Source Field" section of the "Row Status" in the enrollment field mappings to use a custom script.

The custom script essentially tells learn how to properly interpet the status values coming in from Banner in the XML so that users can be marked as unavailable when they drop a course.

The script that we used to make this work (again, this goes in the "Source Field" of the Row Status row in the Enrollment object field mappings) was:

var rs = true;

if ( data.rolesToAddOrUpdate.get(0).status = 0 ) {

rs = false;

}

rs;

This allowed drops to process correctly.

Moving Forward

I expect to have additional issues as we move through the semester, and I'll come back and update periodically.   If you have any useful information or insight based on experience at your own school, please share in the comments.