Subsets and LoadMapping

This page covers some important things to know about working with the active-record like features present on models as well as working with LoadMaps and how they can help you.

Data Subsets on Models

There are various ways you can fetch a subset of some data in ADM. In the Repositories documentation we talk about several methods for doing so. We also talk about setting model constraints in the Model Relationships "Model to Table Rows" section. What isn't discussed elsewhere is how to fetch a subset of a collection when working with models directly.

What do I mean by that? I mean starting with a model (for instance a User) and then fetching some related data from it:

// Starting with a User model...
// Fetch the list of Articles that user has written and count them.
echo $user->articles->count();

Under the hood, repositories still get invoked because Models don't directly interact with the database in ADM, but it's still an important distinction between starting with a model or starting with a repository because the execution flow will be different. When working with repositories you'll probably be calling findAll() or findOne(), and with model relationships you'll be calling the attribute name.

How to Customize a Subset:

You can call any model relationship as a function and pass it a closure that modifies the return results.

Example:

$recentArticles = $user->articles(function($query) {
  return $query->where('published_year', 2018, '>=');
});

Another Example:

// Pretend we got these vars from somewhere...
$currentPage = 0;
$itemsPerPage = 20;

// Let's fetch a paginated view of the articles this user has written:
$currentPageArticles = $user->articles(function($query, $page, $itemsPerPage) {
  return $query->offset($page * $itemsPerPage)
               ->limit($itemsPerPage);
}, [$currentPage, $itemsPerPage]);

LoadMapping

LoadMapping solves several model relationship related problems, and while the simplest examples are easy to understand, some of the more detailed uses can be kinda confusing. Hopefully after reading this you'll understand what a LoadMap is and how it works.

LoadMap Creation

LoadMaps take two arguments as input both of which should be arrays or can be left blank (they are optional). The first array is for mapping fields from a database to the model; the second array is for specifying which relationships should be loaded by default before the results are given back to you.

Before we dive into explaining anything, here's a look at a complicated LoadMap. Don't worry about this looking super confusing, we'll dive into the specifics with the uses cases below then revisit this at the end:

Complicated LoadMap:

$loadMap = new \Cora\Adm\LoadMap([
    'category' => 'type'
  ], [
    'madeBy' => new \Cora\Adm\LoadMap([
      'user_firstName' => 'firstName',
      'user_lastName' => 'lastName',
      'user_id' => 'id'
    ], [
      'Job' => new \Cora\Adm\LoadMap([
        'job_id' => 'id',
        'job_name' => 'name',
        'created_date' => '!created_date'
      ])
    ])
  ],
  function($model, $str) {
    $model->status = $str;
  },
  ['New']
);

Use Case #1: Map Unmatching Fields

Let's say you get a result from a database, but the field names don't match up with what the model is expecting; the first use-case of LoadMaps is mapping arbitrary fields to model attributes. The benefit of this capability doesn't make a lot of sense at first glance as any changes to database field names can be more permanently mapped to the model using the model's "field" descriptor which is what you'll want to use in most situations. Still, understanding how this remapping can be applied is important to understanding how to do efficient model loading.

Let's say you grabbed a collection of Users from the database and you want the "email" field to be mapped to the "username" attribute on the User models. You could do that using a LoadMap like so:

// The first parameter of a LoadMap is for passing an array 
// of mappings.
$loadMap = new \Cora\Adm\LoadMap([
  'email' => 'username'
]);

// Just add the LoadMap as the 3rd argument to our findAll() or findOne() call
$users = $repo->findAll(false, false, $loadMap);

When using models to work with API endpoints, you'll probably want to convert the results into a format such as JSON. The built-in "toJson" method on Models and Collections is handy for that. However, when converting your data to JSON, nothing will be dynamically loaded. So if you need the data about a User's Job sent (and not just the ID that references their job), you have to make sure that Job data is loaded before you call toJson(). In the past this meant iterating through any models in your result and attempting to access the Job data so that dynamic loading would be triggered like so:

$users = $repo->findAll();
foreach($users as $user) {
  // "touch" the job data, even though not doing anything with it here,
  // just to get it to load
  $user->job->name;
}
echo $users->toJson();

This worked, but always felt sideways because of the way you went about forcing data to load. With LoadMaps, you have a better, more clear way of loading the data you need upfront without resorting to wonky loops. LoadMaps take an array as the 2nd argument and allow you to specify relationships you want loaded right away. For example:

// The second parameter of a LoadMap is for passing an array 
// of relationships you want pre-loaded
$loadMap = new \Cora\Adm\LoadMap([], [
  'job' => true
]);

// Just add the LoadMap as the 3rd argument to our findAll() call
$users = $repo->findAll(false, false, $loadMap);

Using the field mapping feature along with the relationship loading specification might look like this:

// The first parameter is for field mappings
// The second parameter is for relationship pre-loading
$loadMap = new \Cora\Adm\LoadMap([
    'email' => 'username'
  ], [
    'job' => true
]);

// Just add the LoadMap as the 3rd argument to our findAll() call
$users = $repo->findAll(false, false, $loadMap);

The above will cause the email field to be loaded into the "username" attribute on the User models and cause the related Job model to be loaded for each User.

Use Case #3: Efficiently Loading Models

Now that we've covered the mapping of attributes and the pre-loading of models, we can talk about how to load additional models efficiently. Doing this requires more manual work on the part of the programmer, and can significantly increase the code required to fetch your models, so I'd advise anyone reading this to refrain from optimizing too early. There's definitely a moderate code readability cost and the decision on whether to optimize should take that into account. However, the performance benefit could be huge.

For this example, let's continue to use the User and Job examples, but with one minor change; so far we've had the name for a job denoted as its "title". Let's change that so Jobs' have a "name" instead of "title". This way we'll have a conflict in naming between a User's name and a Job's name as well as between a Users' ID and a Job's ID. These conflicts will help us understand why you may need to have a LoadMap embedded within another LoadMap.

So what we want to do is fetch a list of Users, but also have their Jobs populated with data. If we were to do this the simple way and just specify we want the Job data loaded as seen below, it would work but it would execute N+1 queries on the database where N equals the number of users you are fetching:

$loadMap = new \Cora\Adm\LoadMap([], [
  'job' => true
]);

// Fetches all the data we need, but does a separate query 
// per User fetched to grab their Job data. So N+1 queries.
$users = $repo->findAll(false, false, $loadMap);

In order reduce the amount of queries hitting the database, that means we need to do some sort of JOIN when we grab the user data and return the Job info along with the rest of the User columns. In addition, the way data gets fetched using PDO in the default MySQL adaptor means you can't have duplicate field names. If you have a "name" column on the Users table and a "name" column on the Jobs table and you do a JOIN, you'll need to alias the Job name or else it will overwrite the User name column when PDO turns the results into a PHP array. So our query ultimately needs to look something like this:

SELECT 
  users.*, 
  job.id as 'job_id', 
  job.name as 'job_name' 
FROM users JOIN jobs ON (users.job = jobs.id);

To accomplish such a join, we'll pass in a custom closure to the findAll() method and either use the query builder or a manually written query like seen below. Keep in mind, the SELECT * FROM users part of the query will be generated by the ORM automatically, so if you go the query building route, you just need to add modifications to that base query:

Using Query Builder:

$users = $repo->findAll(function($query) {
  // Specify the JOIN we need
  $query->join('jobs', [['users.job', '=', 'jobs.id']]);

  // Change the select part from "SELECT *" to something else
  $query->select(["users.*", "job.id as 'job_id'", "job.name as 'job_name'"]);
  return $query;
});

Using Custom Query:

$users = $repo->findAll(function($query) {
  return $query->custom("
    SELECT 
      users.*, 
      job.id as 'job_id', 
      job.name as 'job_name' 
    FROM users JOIN jobs ON (users.job = jobs.id)
  ");
});

That should get us the data we need in ONE query for all our users and their related jobs without the need to do any additional queries. The next step is getting the data populated correctly into the models so that we can do $user->job->name where Job is a proper Job model. To do that we need to turn to the LoadMap functionality we covered in use cases #1 and #2, and then take everything up another notch. Specifically, when we do field mapping like so:

$loadMap = new \Cora\Adm\LoadMap([
    'email' => 'username'
  ], [
    'job' => true
]);

The field mapping above only applies to the loading of User models! We CANNOT do the following:

$loadMap = new \Cora\Adm\LoadMap([
    'email' => 'username',
    'job_id' => 'id',
    'job_name' => 'name'
  ], [
    'job' => true
]);

^ This would NOT WORK like we want. What that would do is override each Users' ID and name with that of their job. To load the Job ID and Name info into the Job model attached to each user, we need to embed a 2nd LoadMap like so:

$loadMap = new \Cora\Adm\LoadMap([
    'email' => 'username'
  ], [
    'job' => new \Cora\Adm\LoadMap([
      'job_id' => 'id',
      'job_name' => 'name'
    ])
  ]
);

Notice that we replaced the boolean true value indicating that we want the Job relationship loaded, and are now defining a LoadMap in that spot. This embedded LoadMap will get passed to the Job model when it gets loaded instead of the ORM doing a new query to fetch the Job info separately. Now it will assume you are defining mappings for it to grab data (which we are).

Important Sidenote:

It's worth noting that in addition to any explicit mapping you define, sub-models (such as Job) will also map any other matching fields to their model too. For example: let's say both Users and Jobs have a "created_date" field. Even though we didn't fetch the Job's created_date in our query, if the User created_date is present, that field will get passed to the Job model and the Job model will assume the field can be loaded for its created_date. To prevent unwanted mappings from happening you have two options: 1. You can fetch the Job created_date in a similar fashion as we did with ID and Name and alias it as job_created_date or something and map that in, OR 2. you can explicitly tell the Job model not to map the created_date field to itself like so (notice the exclamation):

$loadMap = new \Cora\Adm\LoadMap([
    'email' => 'username'
  ], [
    'job' => new \Cora\Adm\LoadMap([
      'job_id' => 'id',
      'job_name' => 'name',
      'created_date' => '!created_date'
    ])
  ]
);

Putting it all together: 

$loadMap = new \Cora\Adm\LoadMap([
    'email' => 'username'
  ], [
    'job' => new \Cora\Adm\LoadMap([
      'job_id' => 'id',
      'job_name' => 'name'
    ])
  ]
);

$users = $repo->findAll(function($query) {
  return $query->custom("
    SELECT 
      users.*, 
      job.id as 'job_id', 
      job.name as 'job_name' 
    FROM users JOIN jobs ON (users.job = jobs.id)
  ");
}, false, $loadMap);

So we should have enough info now to understand in plain English what is happening above. Our LoadMap states that we should load the "email" in as the username on the User models we fetch. Then the 2nd array states that we should load the Job relationship and pass it the LoadMap embedded there, which will map job_id and job_name to the correct fields on the Job model. Finally we're taking that LoadMap and popping it in as the 3rd argument to our repository call which we customized to do a JOIN for the extra data.

One handy thing that can save you a lot of code clutter are on-model LoadMaps if you have a situation where you always want to fetch additional info with a model. See HERE to jump to that section.

Use Case #4: onLoad Function

In addition to mapping fields to the model and controlling which relationships get loaded, another thing LoadMaps can do that you might find useful is dynamically setting properties onLoad. Models have an onLoad() function to which you can hard-code stuff. However, if you're running custom queries, you may find yourself wanting to do stuff onLoad that isn't built into the models. This is where the 3rd and 4th parameters on LoadMaps come in handy.

The 3rd LoadMap argument should be a function that takes a self reference to the model as it's first argument like so:

$loadMap = new \Cora\Adm\LoadMap([], [],
  function($model) {
    $model->status = "new";
  }
);
$users = $repo->findAll(false, false, $loadMap);

In the above example we're setting the status of any models fetched to "new". However, in addition to using the values in the model, you may want to pass variables into the closure. You can do that with the LoadMap's 4th parameter. Example:

$loadMap = new \Cora\Adm\LoadMap([], [],
  function($model, $str1) {
    $model->status = $str1;
  },
  ['new']
);
$users = $repo->findAll(false, false, $loadMap);

If you need to pass multiple variables in, you can do so like this:

$loadMap = new \Cora\Adm\LoadMap([], [],
  function($model, $var1, $var2) {
    $model->status = $var1;
    $model->name   = $var2;
  },
  ['new', 'fooBar']
);
$users = $repo->findAll(false, false, $loadMap);

Complicated Example Revisited

Now that we've covered all the individual pieces of LoadMaps, it's time to revisit the complicated example displayed at the start and make sure you can interpret what it means now:

Complicated LoadMap:

$loadMap = new \Cora\Adm\LoadMap([
    'category' => 'type'
  ], [
    'madeBy' => new \Cora\Adm\LoadMap([
      'user_firstName' => 'firstName',
      'user_lastName' => 'lastName',
      'user_id' => 'id'
    ], [
      'Job' => new \Cora\Adm\LoadMap([
        'job_id' => 'id',
        'job_name' => 'name',
        'created_date' => '!created_date'
      ])
    ])
  ],
  function($model, $str) {
    $model->status = $str;
  },
  ['New']
);

Let's step through this:

Hopefully that explains what's going on. As you can see, the structure is recursive in that you can embed LoadMaps inside other LoadMaps in a multi-level way to accomodate any level of depth you need to handle when deciding what you want loaded on a model and giving you the ability to map variables from your custom JOINed query to model relationships that are buried multiple levels deep in your returned models.

It's probably also clear that this functionality can easily get very complex if you need to load anything deeper than immediate relationships, so use it as you see fit. It's a powerful feature.

Forced Loading Advanced

In one of the sections above we spoke about loading additional related models and how to force certain relationships to be pre-loaded, especially in the context of returning data from an API endpoint. One of the examples given was:

// The second parameter of a LoadMap is for passing an array 
// of relationships you want pre-loaded
$loadMap = new \Cora\Adm\LoadMap([], [
  'job' => true
]);

// Just add the LoadMap as the 3rd argument to our findAll() call
$users = $repo->findAll(false, false, $loadMap);

One thing that's important to know is that the boolean true for loading the "job" relationship as seen in that example is the same as doing:

$loadMap = new \Cora\Adm\LoadMap([], [
  'job' => new \Cora\Adm\LoadMap([], [], true)
]);

That third argument to the LoadMap specifies that the 'job' relationship needs to be fetched on its own using a separate query. Why would you need to specify that? Well, when you define a LoadMap within another LoadMap, there is an assumption made that the main query which will be run will return all the data needed for any SINGULAR nested models. Because of this assumption, dynamic loading is disabled for singular relationships so that the LoadMap can map the data to the model.

I realize that's a mouthful, so let's look at an example that will hopefully make sense of what we're talking about. Notice the line 'role' => new \Cora\Adm\LoadMap() in the following because it's the nested LoadMap we're talking about:

// Create LoadMap
$loadMap = new \Cora\Adm\LoadMap([], [
  'role' => new \Cora\Adm\LoadMap([
    'role_id' => 'id'
  ])
]);

// Custom data fetch to fetch Users with their Role info in one query
$users = $repo->findAll(function($query) {
  $query->join('roles', [['users.role', '=', 'roles.id']])
        ->select(["users.*", "roles.id as 'role_id'", "role.title"]);
  return $query;
}, false, $loadMap);

Because the "role" relationship is defined using a nested LoadMap, ADM assumes we will be doing a customized query with a JOIN (which we are doing in the example). This way, no extra queries are needed to fetch the Role model data, which makes the code super fast.

However, what if each Role had a list of Permissions attached to it, and you wanted those fetched too? Well you can specify you want that additional relationship loaded using the second LoadMap parameter like so:

// Create LoadMap
$loadMap = new \Cora\Adm\LoadMap([], [
  'role' => new \Cora\Adm\LoadMap([
    'role_id' => 'id'
  ], [
    'permissions' => true
  ])
]);

// Custom data fetch to fetch Users with their Role info in one query
$users = $repo->findAll(function($query) {
  $query->join('roles', [['users.role', '=', 'roles.id']])
        ->select(["users.*", "roles.id as 'role_id'", "role.title"]);
  return $query;
}, false, $loadMap);

Now the Users and Roles will be fetched the same as before, but as each Role is loaded the permissions attached to it will also be fetched in a separate query.

Next, what if each Permission had a reference to a group of Users who have that permission and we wanted that User Group loaded? Forcing that info to load would look like this:

// Create LoadMap
$loadMap = new \Cora\Adm\LoadMap([], [
  'role' => new \Cora\Adm\LoadMap([
    'role_id' => 'id'
  ], [
    'permissions' => new \Cora\Adm\LoadMap([], [
      'userGroup' => true
    ], true)
  ])
]);

Finally, what if each User Permission Group had a User who was the designated moderator for some reason? Forcing that Moderator user model to load would look something like:

// Create LoadMap
$loadMap = new \Cora\Adm\LoadMap([], [
  'role' => new \Cora\Adm\LoadMap([
    'role_id' => 'id'
  ], [
    'permissions' => new \Cora\Adm\LoadMap([], [
      'userGroup' => new \Cora\Adm\LoadMap([], [
        'moderator' => true
      ], true)
    ], true)
  ])
]);

Blank Models When Force Loading

What we were doing above with "true" booleans is what I call Forced Loading. You're telling ADM to always load that relationship when present. However, what happens when a relationship you're attempting to force load is null?

This question is especially important when working with APIs because changes in the format of the response can be annoying to deal with. For example, imagine this is the response some front-end javascript received from your API:

[
  {
    name: "User 1",
    userGroup: {
      name: "Moderators"
    }
  },
  {
    name: "User 2",
    userGroup: null
  }
]

If you were to try doing a foreach loop and access the name of the UserGroup the user is in like so: user.userGroup.name, you'll likely encounter a null reference error because the 2nd user doesn't have a group.

When force loading singular models using a LoadMap, ADM will utilize blank models in order to standarize the response you give back if you do an automated toJson() call. In the case of plural relationships such as "permissions", if there are no related permissions, it will return an empty array.

To see how this works let's use a slightly simplified version of the LoadMap we recently defined as seen here:

// Create LoadMap
$loadMap = new \Cora\Adm\LoadMap([], [
  'role' => new \Cora\Adm\LoadMap([], [
    'permissions' => new \Cora\Adm\LoadMap([], [
      'userGroup' => new \Cora\Adm\LoadMap([], [
        'moderator' => true
      ], true)
    ], true)
  ], true)
]);

And let's pretend we're doing code like the following that uses the data:

foreach (users as user) {
  foreach (user.role.permissions as permission) {
    echo permission.userGroup.moderator.name
  }
}

Because of the system ADM employs to load blank models, the above pseudo code should run fine even if there are null values present in the data.

In this way, relationships that you are forcing to load using a LoadMap should be fairly consistent and hopefully not require a ton of work arounds to deal with imperfect data.

On-Model LoadMaps

There may be situations where you ALWAYS want to fetch some additional information with a model. For instance, with your User models, maybe you always want to fetch their primaryRole information. When that happens you can use on-model LoadMaps combined with model constraints to remove the need to manually specify a custom query and LoadMap each time you need to grab those models out of a repository. Below we'll show a manual query and LoadMap (with simple User model), and then show the equivalent functionality using an on-model LoadMap.

Manual LoadMap Example

Model: 

class User extends \Cora\App\Model {
  public $model_attributes = [
    'id' => [
      'type'          => 'int',
      'primaryKey'    => true
    ],
    'email' => [
      'type'  => 'varchar',
      'index' => true
    ],
    'firstName' => [
      'type' => 'varchar'
    ],
    'lastName' => [
      'type' => 'varchar'
    ],
    'primaryRole' => [
      'model' => 'Role'
    ]
  ];
}

Fetching Models From Repository: 

// Define LoadMap
$loadMap = new \Cora\Adm\LoadMap([], [
  'primaryRole' => new \Cora\Adm\LoadMap()
]);

// Fetch models using modified query and passing LoadMap
$users = $repo->findAll(function($query) {
  // Specify the JOIN we need
  return $query->join('roles', [['users.primaryRole', '=', 'roles.role_id']]);
}, false, $loadMap);

On-Model LoadMap Example

The following model definition and repo use is equivalent to the manual example shown above.

Model: 

class User extends \Cora\App\Model {
  public $model_attributes = [
    'id' => [
      'type'          => 'int',
      'primaryKey'    => true
    ],
    'email' => [
      'type'  => 'varchar',
      'index' => true
    ],
    'firstName' => [
      'type' => 'varchar'
    ],
    'lastName' => [
      'type' => 'varchar'
    ],
    'primaryRole' => [
      'model' => 'Role'
    ]
  ];

  public static function model_loadMap() 
  {
    return new \Cora\Adm\LoadMap([], [
      'primaryRole' => new \Cora\Adm\LoadMap()
    ]);
  }

  public static function model_constraints($query) 
  {
    return $query->join('roles', [['users.primaryRole', '=', 'roles.role_id']]);
  }
}

Fetching Models From Repository: 

$users = $repo->findAll();
NOTICE: If you have an on-model LoadMap defined and then pass a custom one to the repo when fetching models, the one passed in will override the on-model version.