xAPI Conformance and Certification Testing Requirements

This is intended to be a listing of technical conformance requirements. We're looking for suggestions that address "What is technically required by the spec?" and "What else should be tested for solid interoperability?"

 

All of the integrated requirements currently listed are from ADL's testing requirements document 

Consistent vocabulary id for xAPI community

xAPI spec. has only guaranteed data structure interoperability. To guarantee semantic interoperability for xAPI data sent by different systems, there should not be multiple ids for one semantic meaning. There are a lot of considerations in vocab. id, including naming appropriateness, naming structure, id persistence, both human-readable and machine-readable required, and CoP management... these are all documented in ADL's publications as below. It seems to me there is no enough awareness about all these, and people just choose whatever others used (for example, what you get when you google tin can api), or just create their own. Even we spent the efforts to work with ADL (Jason Haag, thanks!!!) for building AcrossX vocab. (https://w3id.org/xapi/acrossx) We still need to handle multiple ids issue in Visca (http://www.visualcatch.org/) .

Vocab Spec. from ADL – gitbook: http://bit.ly/read-vocab-spec 

Vocab Primer from ADL – gitbook: http://bit.ly/read-vocab-primer

Guidelines for IRI Design and Persistence

Controlled Vocabulary Considerations for the Experience API (xAPI)

Jessie Chuang

xAPI Chinese CoP Lead

  • Guest
  • Aug 1 2016
  • New
Tags ID, Verb, Activity
  • Attach files
  • Jason Haag commented
    August 11, 2016 14:55

    I don't feel this impacts LRS conformance testing. It does impact learning analytics quality, querying, and reporting. This more in line with best practices, governance, and improving semantic interoperability in general of xAPI data. This requirement does indicate a challenge we currently have with decentralization of xAPI vocabularies managed by CoPs. I will write up a more detailed response with more explanation. For now, I would categorize this requirement as impacting LRS conformance testing. 

  • Jason Haag commented
    August 11, 2016 14:55

    correction: I would NOT categories this requirement as impacting LRS conformance testing.

  • Jason Haag commented
    August 11, 2016 21:22

    I wanted to follow up on this thread with more thoughts on this topic of consistent vocabulary IRIs. This issue has started to rear it’s ugly head more frequently during the past few years. The question being asked here is “Does IRI consistency or duplication impact LRS conformance?”

     

    I don’t believe it impacts our immediate conformance requirements or the LRS tests for the conformance test suite. Please voice your perspectives or opinions if you disagree. Besides impacting the potential for semantic interoperability of data used for things like personalized, adaptive learning, I believe a more immediate impact of this issue is on learning analytics (querying, reporting), xAPI adoption, maturity, and standardization of the spec.

     

    The Activity Streams Vocabulary and IRIs are a real problem mostly because nobody from the xAPI community created them. If any community of practice creates a profile with vocabulary identifiers that happen to unknowingly conflict with Activity Streams identifiers then some LRS developers/consultants have stated this causes a "multiple identities issue.” I'm not sure what this means in detail, but I think it basically means that it makes generating reports and analytics tougher on the LRS or the person writing the queries.

     

    Take a look at these attachments for an example of duplicates: https://drive.google.com/drive/u/0/folders/0BxhK5TH2EspheEM5YXFZaXpfN0E 

     

    We put together two spreadsheets showing duplicate verb IRIs. This problem actually existed long before the ADL vocabulary guidance/practices were published earlier this year. And there are probably many more IRIs out there in the wild that we don't even know about. We conducted a quick comparison of the tincan registry verbs and the verbs published on the ADL vocab server. Which came first? Activity Streams or ADL verbs? The Chicken or the Egg? I’m not sure, but there are many conflicts between those verbs alone. Has this conflict been identified previously? Or are people and LRS implementations just handling it in different ways? If so, the fact that they are handling it differently and rejecting some IRIS but not others is a real threat to interoperability, isn’t it? If this “multiple identities” / IRIs issue is really a problem for reporting then why are there multiple identifiers for several of the Activity Streams IRIs for the exact same ADL verbs (which are pretty much canonical)? Verbs such as: attended, completed, experienced, interacted, satisfied, shared, and terminated are examples of this. These are all ADL verbs duplicated in the Activity Streams vocabulary and both are listed in the TinCan registry.

     

    And also take a look at this document the vocabulary group circulated last October: https://docs.google.com/document/d/1wscWLBX_LmM4BOej5MLMaoJsu2aZvMOxjRGLycLoxlU/edit (excellent comments/thoughts from Andrew Downes and others)

     

    Yet another problem with decentralization is that now even authoring tools are allowing people to create their own verb IRIs on-the-fly within their authoring tool, creating more even more potential problems for duplication. If we all start to embrace existing web standards and practices such as publishing our IRIs as Linked Data it would help enable some really cool things such as allowing authoring tool vendors to provide real-time approved/curated list of terms. We’ve proven this with our vocabulary spec, primer, and research published earlier this year by the vocab working group. While CoPs are starting to apply these practices for vocabulary IRIs, how do we deal with the others?

     

    The core problem is that our vocabulary model and the process of generating IRIs is too decentralized and not heavily curated. Anyone can do whatever they want without a governing process to truly manage and approve it all. The TinCan registry was an early attempt to solve this problem, but there seems to be so much flexibility and widespread adoption of xAPI that it is impossible to enforce using a centralized approach unless it is put directly back into the spec language. How do we check for consistency and quality of IRIs? I could be wrong, but I don’t think this can be enforced as a conformance requirement right now. Perhaps there is a way for a Learning Record Provider (LRP) conformance test of some sort that dereferences the IRIs used in statements to help enforce this practice in the future? Anyhow, I digress…

     

    I believe we need a renewed strategy to centralize all xAPI IRIs. I’m not sure if this would present breaking changes to the spec or not. What are your thoughts?

     

    Moving forward I propose we version out all of the existing Activity Streams identifiers as well as any duplicate identifiers, and have the community come to a consensus on ADL or someone else becoming the single authority for xAPI vocabulary identifiers. In fact, there’s no reason it couldn’t be managed by the community as a GitHub repo with several companies and organizations committing to being a unified “community authority” for maintaining it. Some tools and processes would need to be put in place for this to work. However, how would something like this impact existing LRS implementations? I’m have not built an LRS so it would be good to hear what other LRS implementors/vendors think.

     

    Removing core IRIs from the spec and the general decentralization of IRIs and metadata to CoPs was a good idea at the time, but appears to be causing problems in hindsight. Now we have vendors saying they won't adopt what was in the current spec and are only supporting IRIs of their choosing? I'm afraid our efforts a creating a vocabulary server and publishing tools are going to be futile if we don't first force the community to acknowledge the problem, version the old IRIs to new ones, and accept some type of centralization and formal process around identifier publishing & management moving forward. 

     

    It is going to take a serious community commitment and overarching strategy. It’s nobody’s fault really that this problem exists. The fact is that it is a result of being early adopters of an evolving, quickly adopted brilliant spec. I believe it’s a problem that can be solved if we can come to an agreement.

  • Guest commented
    August 18, 2016 21:04

    Thanks a lot for Jason's comment which is very thorough on this issue.

    I suggest we look back on what we advocate about xAPI and its promise. What does it take to get there really? And we should communicate clearly there are several crucial elements to achieve data interoperability, so what we call "compliance" might have level 1, 2, 3., to get to what xAPI promises.

    This issue might not impact so called LRS compliance, if we define clearly what "LRS compliance" mean - a LRS is only a big data storage, nothing more.(not guarantee data interoperability, and no data analytics)

    As from our experience of providing xAPI analytics, it's a cost that we need to do the mapping and translation for vocab.

    I agree with Jason: 

    I believe we need a renewed strategy to centralize all xAPI IRIs. Moving forward I propose we version out all of the existing Activity Streams identifiers as well as any duplicate identifiers, and have the community come to a consensus on ADL or someone else becoming the single authority for xAPI vocabulary identifiers.

    ADL is currently still the stewardship of xAPI, so it seems the governance needs to come from ADL. But ADL can distribute/delegate the responsibility to appropriate parties such as CoP.(those really hands on very dirty with the mess and know it well)

  • Guest commented
    August 18, 2016 21:06

    Sorry I am the guest in this issue, because I didn't login until submitting stuffs so it shows guest. -- Jessie

  • Gravatar
  • Gravatar
  • Gravatar
  • Gravatar
  • Gravatar
  • Gravatar