Examples

In this section we will provide an example of how the Batch API can be used to synchronize data between Ardoq and an external system. We will use a simple example with just enough complexity to demonstrate the key features of the the Batch API.

Let's imagine that we have following data that we want to keep in sync with Ardoq:

We have a people table:

Name	Email
Alice	alice@my-org.com
Bob	bob@my-org.com
Erik	erik@my-org.com

And a located at table:

Person	Location
alice@my-org.com	Oslo
erik@my-org.com	Oslo

This data has the following properties:

When a person joins the company they will be added to the people table.
When a person leaves the company they will be removed from the people table.
A person may or may not have a location.
If a person moves location the located at table will be updated.
If a person leaves the company they will be removed from both the people table and the located at table.

Our task is to keep the data in Ardoq in sync with the data in the tables above.

Modeling

We decide to model this in Ardoq using a Person and Location component types, and a reference type called "located at".

A Person component will be identified by a custom field called User Email.
A Location component will be identified by its name.

We will assume that the Person and Location components are created in separate workspaces. The "located at" reference is created in the same workspace as the Person components.

Workspace Context

As a user of the API you make requests to the workspace context to get the type information.

First for the workspace with people:

{
  "rootWorkspace": "f2c84727cca064bc0080d0fe",
  "customFields": [
    {
      "apiKey": "user_email",
      "label": "User Email",
      "type": "Email"
    }
  ],
  "componentTypes": [
    {
      "typeId": "p1024822409134",
      "name": "Person",
      "customFields": ["user_email"]
    }
  ],
  "referenceTypes": [
    {
      "type": 1,
      "name": "located at",
      "customFields": []
    }
  ]
}

And then for the workspace with locations:

{
  "rootWorkspace": "6599dbecbc241d0001ccdd42",
  "customFields": [],
  "componentTypes": [
    {
      "typeId": "p1711093589755",
      "name": "Location",
      "customFields": []
    }
  ],
  "referenceTypes": []
}

Integration plan

Now that we have this information we can start to think about the integration.

Essentially we want to ensure that every every Person, Location and "located at" reference in the above tables is reflected in Ardoq and that every component and reference in Ardoq is reflected in the tables above. This two-way synchronization is key to ensuring that the data in Ardoq is kept up to date. If we only synchronize in one direction then we may end up with data in Ardoq that is not in the tables above or vice versa.

This means that the following conditions should be met:

For every row in the people table, a corresponding Person component should be in Ardoq. This ensures that Ardoq is not missing any people.
For every unique location in the located at table, a corresponding Location component should be in Ardoq. This ensures that Ardoq is not missing any locations.
For every row in the located at table, a corresponding "located at" reference should be in Ardoq. This ensures that Ardoq is not missing any references.
For every Person component in Ardoq, a corresponding row should be in the people table. This ensures that Ardoq does not retain any people that are no longer in the people table.
For every Location component in Ardoq, a corresponding row should be in the located at table. This ensures that Ardoq does not retain any locations that are no longer in the located at table. You may decide that you would like to retain locations in Ardoq even if they are not in the located at table. For example, you may manually create a Location component in the Ardoq App. Even if the location is not in the located at table you may still want to retain it in Ardoq.
For every located at reference in Ardoq, a corresponding row should be in the located at table. This ensures that Ardoq does not retain any references that are no longer in the located at table. You may decide that you would like to retain references in Ardoq even if they are not in the located at table.

Our task it to translate these requirements into one (or more) batch requests that can be sent to the API.

Single Request

We can create a single batch request that will ensure that the data in Ardoq is in sync with the data in the table above.

The key here is to use a combination of upsert and delete operations as well as Batch Ids to ensure that the data is in sync.

Deleting by match allows us to delete any components and references that are not required by the payload. For components we match by the typeId of the component and the rootWorkspace that the component belongs to. This ensures that we only delete the components that we want to delete as these are the types that the payload manages. For references we match by the type of the reference and the workspaces that the source and target components belong to.

Warning

Deleting my match is a powerful feature that should be used with caution. It is possible to delete components that are not intended to be deleted if the match is too broad.

In the above example we have deleted references based on their type and the workspaces that their source and target components belong to. There is an assumption here that the "located at" reference is only used to link Person and Location components in these two workspaces. If you had say a an Office component in the same workspace as the Person components and linked your Office components to your Location components using the same reference type then you would delete these references as well.

{
  "components": {
    "delete": [
      {"match": {"rootWorkspace": "f2c84727cca064bc0080d0fe", "typeId": "p1024822409134"}},
      {"match": {"rootWorkspace": "6599dbecbc241d0001ccdd42", "typeId": "p1711093589755"}},
    ],
    "upsert": [
      {
        "batchId": "Person:alice@my-org.com",
        "body": {
          "rootWorkspace": "f2c84727cca064bc0080d0fe",
          "typeId": "p1024822409134",
          "name": "Alice",
          "customFields": {"email": "alice@my-org.com"}
        }
      },
      {
        "batchId": "Person:bob@my-org.com",
        "body": {
          "rootWorkspace": "f2c84727cca064bc0080d0fe",
          "typeId": "p1024822409134",
          "name": "Bob",
          "customFields": {"email": "bob@my-org.com"}
        }
      },
      {
        "batchId": "Person:erik@my-org.com",
        "body": {
          "rootWorkspace": "f2c84727cca064bc0080d0fe",
          "typeId": "p1024822409134",
          "name": "Erik",
          "customFields": {"email": "erik@my-org.com"}
        }
      },
      {
        "batchId": "Location:Oslo",
        "body": {
          "rootWorkspace": "6599dbecbc241d0001ccdd42",
          "typeId": "p1711093589755",
          "name": "Oslo",          
        }
      },
      {
        "batchId": "Location:London",
        "body": {
          "rootWorkspace": "6599dbecbc241d0001ccdd42",
          "typeId": "p1711093589755",
          "name": "London",
        }
      },
    ],  
  "references": {
    "delete": [
      {"match": {"rootWorkspace": "f2c84727cca064bc0080d0fe", 
                 "targetWorkspace": "6599dbecbc241d0001ccdd42", 
                 "type": 1}},
    ],    
    "upsert": [
      {
        "uniqueBy": ["source", "type"],        
        "body": {
          "type": 1,
          "source": "Person:alice@my-org.com",
          "target": "Location:Oslo"
        }
      },
      {
        "uniqueBy": ["source", "type"],        
        "body": {
          "type": 1,
          "source": "Person:bob@my-org.com",
          "target": "Location:London"
        }
      },      
      {
        "uniqueBy": ["source", "type"],        
        "body": {
          "type": 1,
          "source": "Person:erik@my-org.com",
          "target": "Location:Oslo"
        }
      },      
    ]
  }
}

As this payload represent the total state of the data in the table above, it can be sent to the API at regular intervals to ensure that the data in Ardoq is kept in sync, even if the data in the table changes.

Multiple Requests

As the size of your data grows you may find that the payload becomes too large to send in a single request. In this case you can split the payload into multiple requests.

Recommendation

We recommend that you partition your components by a combination of rootWorkspace and typeId.

First we upsert the Person components.

{
  "components": {
    "delete": [
      {"match": {"rootWorkspace": "f2c84727cca064bc0080d0fe", "typeId": "p1024822409134"}}      
    ],
    "upsert": [
      {
        "batchId": "Person:alice@my-org.com",
        "body": {
          "rootWorkspace": "f2c84727cca064bc0080d0fe",
          "typeId": "p1024822409134",
          "name": "Alice",
          "customFields": {"email": "alice@my-org.com"}
        }
      },
      {
        "batchId": "Person:bob@my-org.com",
        "body": {
          "rootWorkspace": "f2c84727cca064bc0080d0fe",
          "typeId": "p1024822409134",
          "name": "Bob",
          "customFields": {"email": "bob@my-org.com"}
        }
      },
      {
        "batchId": "Person:erik@my-org.com",
        "body": {
          "rootWorkspace": "f2c84727cca064bc0080d0fe",
          "typeId": "p1024822409134",
          "name": "Erik",
          "customFields": {"email": "erik@my-org.com"}
        }
      }      
    ]  
}

We then upsert the Location components.

Finally we upsert the references. href="#__codelineno-4-1">{ "components": { "delete": [ {"match": {"rootWorkspace": "6599dbecbc241d0001ccdd42", "typeId": "p9999999999999"}}, ], "upsert": [ { "batchId": "Location:Oslo", "body": { "rootWorkspace": "6599dbecbc241d0001ccdd42", "typeId": "p9999999999999", "name": "Oslo", } }, { "batchId": "Location:London", "body": { "rootWorkspace": "6599dbecbc241d0001ccdd42", "typeId": "p9999999999999", "name": "London", } }, ] }

Recommendation

Currently the Batch API does not support matching references for deletion by the type of the source and target components.

We recommend that you partition your references by a combination of rootWorkspace, targetWorkspace and typeId. If for example you had modelled Office components to be in the same workspace as you People components and linked them to Location components using the same reference type "located at", then you would include the references between Office and Location components in the same payload as the references between Person and Location components. This would ensure that you did not delete the references between Office and Location components.

{
  "aliases": {
    "components": {
      "Person:alice@my-org.com": {"customFields": {"email": "alice@my-org.com"}, "rootWorkspace": "f2c84727cca064bc0080d0fe"},
      "Person:bob@my-org.com": {"customFields": {"email": "bob@my-org.com"}, "rootWorkspace": "f2c84727cca064bc0080d0fe"},
      "Person:erik@my-org.com":  {"customFields": {"email": "erik@my-org.com"}, "rootWorkspace": "f2c84727cca064bc0080d0fe"},
      "Location:Oslo":  {"name": "Oslo", "rootWorkspace": "6599dbecbc241d0001ccdd42"},
      "Location:London":  {"name": "London", "rootWorkspace": "6599dbecbc241d0001ccdd42"}      
    }
  },
  "references": {
    "delete": [
      {"match": {"rootWorkspace": "f2c84727cca064bc0080d0fe", 
                 "targetWorkspace": "6599dbecbc241d0001ccdd42", 
                 "type": 1}},
    ],    
    "upsert": [
      {
        "uniqueBy": ["source", "type"],        
        "body": {
          "type": 1,          
          "source": "Person:alice@my-org.com",
          "target": "Location:Oslo"
        }
      },
      {
        "uniqueBy": ["source", "type"],        
        "body": {
          "type": 1,          
          "source": "Person:bob@my-org.com",
          "target": "Location:London"
        }
      },      
      {
        "uniqueBy": ["source", "type"],        
        "body": {
          "type": 1,          
          "source": "Person:erik@my-org.com",
          "target": "Location:Oslo"
        }
      },      
    ]
  }
}