#assessment — Public Fediverse posts on home.social

Mark Carrigan @[email protected] · 2026-05-28 · 10:45 UTC

The second wave of the AI and assessment crisis

In this paper Thomas Corbin, Sue Sharpe & Phillip Dawson suggest that wearable AI will bring a second wave of the assessment crisis. In the first wave, there has been a reliance on the idea that physical examination provides a backstop which can underwrite authenticity: “the physical exclusion of technology at the point of performance” (pg 1). They argue that wearable AI will make it vastly more difficult to enact that exclusion because they can provide real-time cognitive assistant without external markers which indicate they are being used for this purpose.

This is still a new field but it is rapidly growing. Meta sold 7 million smart glasses last year, with signs suggesting growth is accelerating. These are just manufacturer within a broader field of wearable AI that is receiving huge investment. So while someone might be able to spot Meta’s Ray-Ban glasses it’s unfeasible to imagine that every wearable device could be reliably spotted. There also equity issues which arise from the fact these serve real assistive functions for many users: they are dual use in a way which precludes ethical exclusion. The assumption we would ratchet up oversight in order to prevent them being brought into invigilated spaces raises all manner of ethical, legal and political questions. As they put it, “A regime that extends scrutiny further than simply glasses must decide how far into the student’s embodied presentation it is willing to reach” (pg 13). A commitment to excluding these devices necessitates a form of “bodily adjudication” based on two conditions which are decreasingly tenable. From pg 12:

First, invigilators must be able to identify which objects on a student’s person are relevant candidates for scrutiny. Secondly, they must then be able to determine whether those objects are AI-enabled or not. Under conditions of wearable AI, neither condition can be assumed. The issue is not simply that smart glasses may be difficult to distinguish from ordinary eyewear. Rather, it is that the relevant class of wearable technologies no longer maps neatly onto a small set of visibly exceptional devices.

The deeper transition they are pointing to here involves a shift from AI as a discrete tool to one which is embedded in practice in a way that might not ultimately be separable. In this sense I think we can see inline automation tools (Copilot 365 and Grammarly etc) which offer ambient assistant to users as another vector of this transition. I thought this was really important on pg 6-7:

Screen-based AI is structurally different. Consulting ChatGPT on a laptop or smartphone requires directing attention away from the task at hand, engaging with a separate interface, reading a response, and returning to the task. Even when this process becomes routine, it remains episodic. The tool cannot become phenomenologically transparent because the architecture of use requires repeated explicit engagement with a separate object. The user must turn to it, attend to it, and return from it. Smart glasses differ because they operate within, rather than alongside, ongoing activity. They have the architectural capacity to become phenomenologically trans- parent, to withdraw from user awareness and become part of the structure through

The episodic character of user-model interaction for chatbots is exactly what makes meta-cognition possible. They demand articulation, even if minimally, while also making the interaction itself available as an object of reflection that can inform that articulation. This is why it’s possible to use chatbots in an active way. In contrast inline automation tools insert themselves into the flow of activity in a manner which is intended to render this episodic experience unnecessary. This is literally baked into the metaphor of the Copilot. It’s possible to meta-reflect while you’re in flow but I don’t think it’s possible for learners to do this: the space is crucial for developing this capabilities in the first place. For this reason I’d suggest we see the second wave of the assessment crisis as responding to three factors: (a) the declining burden of articulation in chatbots* (b) the parallel growth of inline automation tools (c) the rise of AI wearables. This is how they describe the distinction between the first wave and the second wave. From pg 9

The first wave, exemplified by screen-based systems such as ChatGPT,
created a crisis of practice within an intact institutional framework. Tasks had to be redesigned, expectations renegotiated, and academic integrity policies rewritten, but the basic shape of the problem remained familiar: students were using an external tool, that tool produced identifiable outputs, and institutions could still, with effort, separate students from the tool at particular assessment events. The first wave was a harder version of a problem assessment had encountered before.
The second wave is different in three ways, and each of them matters inde pendently. First, the property itself is structurally new. Screen-based AI is episodic by architecture. The user must turn to it, attend to it, read its response, and return. Even a heavily reliant user is engaging with the tool as a discrete object on discrete occasions. Wearable AI, as the previous sections have argued, has the structural conditions for incorporation. It does not function as a tool the user consults but as a capability the user inhabits. This is not a difference of degree. It is a difference in the kind of relationship a user can have with the technology, and it is not a difference any previous educational technology has had to confront at scale.

Once AI use is no longer “external, episodic, and, at least in principle, distinguishable from the student’s own ongoing activity” (pg 10) then assessment strategies built around exclusion become fundamentally untenable. It’s another argument that supports the notion that fundamental assessment reform has to happen so we might as well get on with it. The problem is that I still don’t believe that processual assessment is adequately scalable within mass higher education. So the vice tightens 😬

*This is what my book with Milan Sturmer is essentially about. The short-form version of the argument is that post-training has made chatbots vastly more able to infer user expectations without deliberate and expansive prompting. Therefore the user has to articulate themselves to get what they want.

#AI #assessment #higherEducation #metaReflection #universities #wearableAI #wearableTechnology

#ai #assessment #highereducation #metareflection #universities #wearableai

Mark Carrigan @[email protected] · 2026-05-27 · 15:46 UTC

What is the problem to which cognitive outsourcing is the solution?

This paper by Thomas Corbin et al reports on a pilot study of philosophy undergraduates exploring their use of AI-reading tools. Their analysis of half of students using generative AI tools in some way for reading. Interestingly, the vast majority (79.1%) recognised the importance of this reading while also citing limited time (65.7%) and intellectually difficulty (33.3%) with the texts. They suggest a positive trend underlying the familiar fears about cognitive outsourcing. From pg 6:

The strong positive sentiment toward GenAI availability (76.2%) suggests these tools are making students more comfortable with challenging content, potentially lowering anxiety barriers to engagement with complex reading material. By providing alternative entry points to challenging texts, GenAI tools may help democratise access, particularly for students who face epistemic barriers to traditional engagement with reading materials. However, this optimistic interpretation must be balanced against potential risks. While GenAI may help students overcome initial barriers, over-reliance on AI-generated summaries could potentially impede the development of critical reading and interpretive skills that are essential to philosophical education.

This is what I mean about the need to respond diagnostically to student AI use. There are real problems in teaching and learning being surfaced by developing trends in student AI-use. What is the problem to which cognitive outsourcing is the solution for students? In asking this question it becomes possible to diagnose the underlying challenges which pre-existed generative AI, as well as to better understand student use in a manner which enables us to steer them towards active rather than passive use of AI.

#AI #assessment #cognitiveOutsourcing #literature #malpractice #readings

#ai #assessment #cognitiveoutsourcing #literature #malpractice #readings

Mark Carrigan @[email protected] · 2026-05-27 · 15:21 UTC

We need structural changes to assessment rather than discursive changes

This is the slightly overstated thesis of this paper. It rests on what I think is a genuinely useful distinction between discursive and structural changes to assessment:

Modifications that rely solely on the communication of instructions, rules, or guidelines to students, such that their success depends entirely on student awareness, understanding, and voluntary compliance with these communica- tions. These changes leave the underlying structure and mechanics of the assessment task unchanged, focusing instead on specifying how students should approach or complete the task.
1091

Modifications that directly alter the nature, format, or mechanics of how a task must be completed, such that the success of these changes is not reliant on the student’s understanding, interpretation, or compliance with instructions. Instead, these changes reshape the underlying framework of the task, constraining or opening the student’s approach in ways that are built into the assessment itself.
1092-1093

The traffic light systems, the 4/5 point AI assessment scale (AIAS) and declarations all constitute discursive approaches in that they fundamentally change how we communicate about assessment. There are three problems which the authors identify with these approaches:

They assume student understanding when the application of abstract categories to real world practice will always be ambiguous, particularly when those categories are formatted at the level of abstraction necessary for a large multidisciplinary university.
They assume student voluntary compliance with the approach, in spite of significant incentives to non-compliance and the aforementioned ambiguity about what constitutes compliance.
They assume student compliance can be meaningfully assessed when there is not really any mechanism through which to do this.

In contrast structural changes actually modify the assessment “by creating conditions where inappropriate AI use becomes difficult or impossible” (1093). These changes can vary but effect ones involve a move from product to process, as well as designing interconnections between assessments such that “the validity of assessment comes not from any single component but from the coherent demonstration of learning across multiple appropriately designed touch points unfortunately” (1095).

The obvious problem that I’m abundantly familiar with as someone who ran a large PGT programme is that it is extremely hard to scale processual assessment. In large cohorts you need to resort to digital platforms in order to do it, which mitigates exactly the assessment security that processual assessment is supposed to provided. This is clearly the way to go in a perfect environment: processual assessment strategy with a healthy dose of authentic tasks and well-designed group would go some way to solving the problems we are no encountering. But I remain unconvinced you can do this reliably in any environment other than, say, the Oxbridge system. The class sizes have to be small and the teacher/student ratio has to be healthy with stable relationships between them. Otherwise it breaks down.

I say that I think this thesis is overstated because it’s not clear to me that discursive changes are necessarily toothless. Firstly, if we assume that the majority of students start from the position of wanting to learn and to follow the rules (two different things) then clarifying expectations is inherently valuable. It provides students with guidance about how to ensure they are learning and to ensure they are not engaged in malpractice. The fact the sector has been crap at doing this doesn’t license the weird dismissiveness in the paper towards clarifying expectations. Secondly, once we have clarified those expectations it becomes possible to have malpractice processes which are more targeted and fine grained. It doesn’t solve the problem but it seems to me inherently better than not having the discursive shift in the first place.

I think their assumption is that assessment structural shift has to happen so why not start now? As they put it on 1096:

The time invested in developing and implementing these discursive approaches is time that could otherwise be used to consider structural changes that will actually work to ensure assessment validity as well as the veracity and reputation of our degrees. When assessment validity hinges on student compliance with unenforceable rules rather than on inherent assessment design, we build educational systems on foundations of sand. Long term solutions require fun- damentally rethinking how assessments are structured rather than how they are explained.

I’m somewhat sympathetic to this view but I also think it’s such a long term process, in such a resource-constrained environment, that we do seriously risk a complete collapse of trust in credentials before then. So how do we undertake discursive approaches (adapting to AI in my terms) while still working towards structural changes (integrating AI in my terms)? How do we stop the former crowding out the space for the latter? The way they describe the two-lane approach opens up a framework for thinking institutionally about how that might be possible. From 1095

One immediate benefit of adopting this structural perspective is that it provides a clearer lens for evaluating emerging institutional frameworks, such as the university of sydney’s ‘two-lane approach’ (Liu and Bridgeman 2023). This framework distinguishes between ‘secure’ (Lane 1) assess- ments which are conducted in-person with controlled conditions, and ‘Open’ (Lane 2) assessments where AI use is uncontrolled (Tertiary Education Quality and standards Agency 2024, p. 51). The structural/discursive distinction we propose offers a potentially useful lens for understanding and extending the efficacy of such approaches. While Lane 1 assessments incorporate structural ele- ments by creating environments where inappropriate AI use is physically restricted, the effectiveness of Lane 2 assessments depends on how they are designed structurally, as simply designating an assessment as ‘Open’ without reconsidering its structural mechanics perpetuates the enforcement illusion we have identified. The most effective implementations of dual-track approaches such as these will therefore be those that recognise the need for structural reconsideration of assessment design in both lanes, albeit in different ways.

#AI #assessment #assessmentIntegrity #higherEducation #malpractice

#ai #assessment #assessmentintegrity #highereducation #malpractice

Mark Carrigan @[email protected] · 2026-05-27 · 14:38 UTC

There is no solution to the AI and assessment problem

This is the core message of a surprisingly upbeat paper. There is no solution to the AI and assessment problem because it’s a classic example of a wicked problem. This means that, as they put it on pg 2:

Wicked problems, as opposed to ‘tame’ problems, do not have ‘correct’ or ‘incorrect’ solutions (Rittel and Webber 1973). This does not mean there are no ways forward, nor does it mean that all ways forward are equally valuable. However, it does mean that responses must look very different. For one, they require a shift from seeking definitive answers to engaging in ongoing, adaptive work shaped by competing priorities and evolving conditions.

There are a number of reasons they claim it is a wicked problem:

It cannot be clearly or conclusively defined
There is no clear criteria for knowing when ‘the solution’ has been reached
There are only better or worse options involving trade offs
There is a lack of clear metrics to adjudicate between these better or worse options
They cannot be studied through trial and error because every trial has real world consequences which means decision makers are on the line for them
The range of putative solutions and potential approaches is pretty much limitless
They exist because of deeper structural issues and reflect these issues
The framing determines which approaches show up for us as relevant

This means academics are “put in the position of needing to make continuous professional judgments in conditions of permanent uncertainty” (pg 12). This is not a good position to be in and it’s not going away. Rather than a council of despair, recognising the character of wicked problems is necessary for helping us cope with being placed in that position:

“First, it lifts the impossible burden on teachers and institutions to immediately get things right once and for all. When problems are unsolvable and ever-changing, missteps and course corrections are not failures. They are part of doing the work well.” (pg 12)
“Alternatively, a wicked problem frame suggests that trade-off are necessary and there is no optimal balance nor solution. The teacher who wondered ‘Have I struck the right balance? I don’t know’ (T6) was describing the uncertainty inherent in weighing pedagogical goals against workload, security against authenticity, current needs against future preparation.” (pg 13)
“Permission to diverge recognizes that in wicked problems, context determines every- thing. What transforms learning in a 20-student philosophy seminar becomes logis- tically impossible with 250 business students. What prepares future lawyers for AI-integrated practice might undermine the clinical skills nurses need.” (pg 13)

It means we can accept there is no fix but rather iterative and evaluative design work which is necessary because the environment has shifted in a fundamental sense. What matters is that we are moving in the ‘right’ direction while ensuring that we build up a variegated (and always provisional) sense of what ‘right’ is that reflects the range of different practices and imperatives within a multidisciplinary university.

#AI #assessment #higherEducation #LLMs

#ai #assessment #highereducation #llms

Dr. Halley Pontes :verified: @[email protected] · 2026-05-27 · 11:02 UTC

Our latest study on work addiction has now been published in the journal Assessment. Congratulations to Stephanie Towch and thanks to Dr. Paweł Atroszko for the collaboration!

Full study available from here: https://doi.org/10.1177/10731911261447068

#workaddiction #assessment #psychometrics #IWAS

#workaddiction #assessment #psychometrics #iwas

Bytes Europe @[email protected] · 2026-05-27 · 09:31 UTC

Providing an evidence base on climate risk | Met Office https://www.byteseu.com/2057016/ #adapted #applied #assessment #birmingham #CCC #CCRA4 #change #Climate #ClimateChange #Fourth #GlobalWarming #IA #independent #office #Professor #report #Risk #Science #Technical #UK #university #well

#well #university #uk #technical #science #risk

Mark Carrigan @[email protected] · 2026-05-26 · 13:48 UTC

What does it mean for students to use AI in active rather than passive ways?

If anyone is wondering why I’ve suddenly started saying ‘AI’ it’s because I’ve (reluctantly) accepted this is a necessary requirement for communicating effectively in higher education policy work. I still think we should be talking about models and will continue to write about them in my theoretical work.

What does it mean for students to use AI in active rather than passive ways? In Generative AI for Academics I talked about the difference between thinking with AI and using AI as a substitute for thinking. This roughly maps onto the cognitive outsourcing concept which I’ve argued we need to move away from. It’s too binary a distinction to capture the complexity of how users engage with AI, even if it does nonetheless track a meaningful distinction which matters. In some cases a user is actively thinking about their use and in other cases they are not. Furthermore, this is a distinction in practice which matters in principle. What it means to use AI is different if you are thinking about the use you are making of it. It doesn’t necessarily mitigate the risks but ceteris paribus it’s better to think about your use then not think about it.

I’ve tried two routes towards fleshing out this distinction as a spectrum. The first is to look at specific practices which a student might engage with in relation to AI. For example the HEPI (2025) research shows a variegated picture in terms of what students have used AI for in assessments. I’ve argued these practices range from the obviously problematic (use in assessment without editing) or obviously acceptable (explain concepts*) but that most are an ambiguous middle-ground in which context-sensitive judgements have to be made in terms of cohort characteristics, disciplinary standards and assessment design. This helps crack open the black-box of AI practice (treating AI use as if it’s fundamentally interchangeable rather intensely varied) but it doesn’t really address the question of what active use actually is. It simply restates the problem at a more granular level of specific practices which we can either assume to be active or passive in all usual cases or which we can inquire about activity or passivity in context-sensitive terms.

The other strategy was to use this notion from Jonathon Jackson’s interesting account of degrees of LLM use in learning. He suggests we need to design learning activities which inculcate the habit of shifting left so that if students reach human-in-the-loop or llm-centric use then they do not remain there. This feels important to me because it highlights how active use is something which has to be worked at longitudinally. It suggests that if we incorporate AI into learning we should do so in a way which ensures a left-shift is likely. This is particularly important when we consider the structural drivers of habituation which are going to intensify in consumer-focused subscription based LLMs over the coming years. If the student is not going to opt out entirely (and obviously they can’t if we’re building this into an assessment) then what matters becomes developing the inclination to pull back into more active forms of use.

In neither case have I really addressed the question though. What is active use? The notion of epistemic agency (introduced to me by Peter Kahn) offers a way through which we might begin to think about this question. Juuso Henrik Nieminen, Eeva Haataja & Peter J. Cobb offer hints of a potential answer in this paper. They define this for students as “their sense of agency in using, evaluating and producing knowledge” (970). It’s the outcome of “students’ transformational relationship with knowledge” (972) facilitated (or frustrated) by the environment in which teaching and learning is taking place. In a case study of student epistemic agency in authentic assessment they define the following area of focus for their inquiry (my bold):

We first focused on students’ accounts of their epistemic actions: how they explained learning and studying as they progressed in the course. We then analysed how these actions reflected students’ orientation to knowledge: how they positioned themselves with respect to knowledge in digitally-mediated authentic assessment
Pg 977

Note that the first concept is emic: how do students account for their learning and progress. The second concept is etic: what can we infer from their actions about their orientations towards knowledge? This split is important I think because it enables us to take student narrations of knowledge work seriously without taking them literally. There’s a further level of inference we can make. Therefore we might ask in relation to AI use:

How do students account for their actions with AI in terms of knowledge?
What can we infer from student actions about their orientation towards knowledge?

In their analysis they offer a number of themes which can help us clarify what to look for in relation to these questions:

A sense of being an active learner
A sense of being a user of knowledge
A sense of contributing to society

These are all things we can ask students about in their use of AI. To what extent do they feel they are using it an active way? It’s a fallible guide but we can nonetheless talk to students about whether it feels like they are learning (thanks to David Meecham for this point). It’s a phenomenological datapoint that can be taken seriously. Likewise we can ask them about the extent to which they feel they are actively engage with knowledge when they use AI? Does it feel like they are passive recipients or that they are linking thinks together in active ways? An interesting point in the paper was the role of interdisciplinarity and the acts of synthesis it invites in bringing this about.

In another paper Juuso Henrik Nieminen and Laura Ketonen talk about the same concept of epistemic agency in terms of assessment more broadly. They argue that what I think of as the promissory function of assessment (ensuring that a student given a credential has the learning the credential claims) and the stakes for students of the ensuing culture undermines an active and transformative engagement with knowledge. If it’s all about validating knowledge conceived of as a property of the individual student then the active engagement (facilitated by the environment) will tend to be neglected. Likewise a focus on employability skills too easily leads to a focus on discrete competencies to be reproduced in the workplace rather than the more diffuse meta-competency (?) which might or might not underlie them. If assessment is targeted at demonstrations of knowing rather than knowing it leaves us with a performative assessment culture.

It’s important that epistemic agency is conceived of in terms of the environment which facilitates or frustrates it. We encounter active or passive use of AI at the level of the individual student and the specific practices they are engaged in. I suggested in the previous post that we might see cognitive ownership at the task level and the learning journey level. The tasks which constitute the student’s learning (including informal learning) jointly combine into a learning journey which is characterised by a certain degree of cognitive ownership. What I’m talking about as cognitive ownership maps onto the phenomenological sense of being an active learners and being a user of knowledge. These are present at the task level and they jointly combine into characteristics of the learning journey.

The evaluative level which the student cannot conclusively adjudicate on is whether a sense of (actual) cognitive ownership is matched by (real) epistemic agency. It’s the latter question which forces us to look again at the context. To what extent is the learning environment (encompassing everything from learning design through to assessment and institutional provision of resources) facilitating epistemic agency? We’ve already seen from the second paper how assessment culture can frustrate epistemic agency at the learning journey level even if it might flicker into being at the task level. This gives us a framework for thinking about institutions as enabling epistemic agency by making it easier for students to use AI in active ways defined by cognitive ownership. It means we need to design environments that make this easier, as well as supporting activities and assessments which make this easier.

So what does it mean to talk about a student using AI in an active way? This is what I’m gesturing towards though it is still provisional:

An experienced sense of being an active learner
An experienced sense of actively working with knowledge
A transformative engagement to knowledge (Nieminen and Ketonen) i.e. the student’s understanding is changed by the interaction
The capability is retained in spite of the AI use (Pritchard’s challenge here)

I think this use is possible. In future posts I’ll have a go at defining it in concrete terms with examples. It’s a high threshold though: it’s ok if not all use meets this threshold but that’s exactly why we should left-shift in Jackson’s terms. It also means we should discourage use which does not incline towards this threshold because that would be ‘cognitive outsourcing’ in precisely the sense in which so many academics are worrying about it.

*The one pushback I had to this was that ‘explaining concepts’ is a problem because of the anglocentric bias of the corpus. Surely this would suggest though that ‘explaining concepts’ using resources from a library or articles from a journal system that hasn’t been colonised would be equally problematic? It seems like a category error to treat this as a problem specific to LLMs (as opposed to other knowledge sources) but I can see the specific risk that LLMs launder objectivity by presenting themselves as authoritative new sources of neutrality. But this itself suggests to me we need to scaffold the practice for students rather than retreat from it.

#AI #assessment #epistemicAgency #learning #passive #pedagogy

#ai #assessment #epistemicagency #learning #passive #pedagogy

Mark Carrigan @[email protected] · 2026-05-25 · 17:29 UTC

Constructive alignment and AI-integration into teaching and learning

In Rethinking the Integration of AI in Higher Education Teaching and Learning, Lilian Schofielda and Xue Zhou consider what AI integration into the curriculum looks like in practice at a module level. They advocate “a structured process that enables educators to systematically align AI tools with AI literacy, linking the literacy process to learning objectives, class activities and assessment methods” (pg 2). This works from the principle of constructive alignment that there should be purposive integration between ILOs, teaching activities and assessment tasks which facilitates a scaffolded and coherent learning journey: the student reaches the capabilities and knowledge defined by the ILOs, in a manner robustly assessed by the assessment tasks and developed through the teaching and learning activity which takes place on the unit. What’s lacking is “simplified, practical guidance that shows educators how to integrate AI literacy holistically in the full cycle of curriculum design, aligning AI literacy with learning outcomes, class activities, and assessment” (pg 3). This is particularly problematic because of the number of educators with relatively low technological pedagogical knowledge (TPK) which makes AI-integration more challenging.

So what is AI-integration? It’s a term I use a lot and this paper challenged me to offer a more substantive definition for teaching and learning. I would argue that AI integration is the purposive incorporation of ‘AI’ into teaching and learning, working towards constructive alignment between ILOs, teaching activities and assessment tasks. I say ‘working towards’ because I think few modules are likely to reach the threshold of full constructive alignment. Indeed the pace of change (technologically and institutionally) means this might be too high a threshold for us to work with for AI-integration for the time being, which means we might think of AI-integration in the thick sense as the destination we are working towards. There can be purposive incorporation of AI into ILOs, teaching activities and assessment tasks in a more piecemeal way but full constructive alignment between them might be an outcome to be enacted over a number of design cycles. I think we should make our peace with this, as long as the incorporation is purposive and it’s informed by sufficient AI literacy (or TPK to use this jargon) to avoid the intellectual and pedagogical risks that might otherwise come from piecemeal integration. This does strain against the idea of constructive alignment however because once you talk about incorporating an objective without an associated learning activity, or a learning activity that doesn’t meet an objective (etc), there’s obvious incoherence which enters into the learning design. But at a practical institutional level arguing that AI-integration can’t take place unless it’s fully constructively aligned is a recipe for preventing AI-integration or keeping it confined to shadow practice in the classroom that doesn’t find its way into a formal module specification. As they observe on pg 11: “constructive alignment emphasises a structured and linear alignment of learning outcomes, teaching activities, and assessment methods, potentially limiting the adaptability required by rapidly evolving AI technologies.“

It follows from this that diffusion of AI in a university doesn’t make integration happen. Integration in the sense I’m advocating here isn’t simply a matter of increasing the quantity of use which takes place in an organisation. It’s the purposeful integration of the technology into organisational activities in a manner which ensures alignment between goals, activities and measurements. In this sense I think we could actually think of something like constructive alignment when thinking of AI-integration more broadly, though I’m just talking about teaching and learning here. Indeed in teaching and learning, diffusion could actually reduce AI-integration by making purposeful integration harder. If it drops into a population with low TPK then it will tend to produce unpurposive, under-literate incorporation by staff, meeting whatever happens amongst the student population. If actual classroom practice is swamped by chaotic individualized use of AI then it constitutes a challenge which needs to be undone on the teacher and/or student side before meaningful integration can occur. It means that institutionally supported diffusion can actually intensify the problem of adapting to AI (as a fact on the ground) which needs to be under some sort of control before integrating AI (as a purposive undertaking) becomes possible. As they observe on pg 5, “educators face apprehension and resistance to AI integration due to concerns about existing academic misconduct, ethical considerations, as well as the assumption that AI subverts learning”. This is exactly what I mean when I talk in my work about the tension between adapting to AI and integrating AI.

This is the practical model they call the GenAI Curriculum Alignment Model (pg 6):

1. Define intended learning outcomes

2. Design assessments that measure student achievement of these outcomes

3. Select appropriate AI tools

4. Develop teaching activities that enable students to meet the learning outcomes

5. Continuously monitor and evaluate AI integration to refine educational practices.

While I remain sceptical that learning design models as a whole simply reflect and/or guide teaching and learning practice (as opposed to constituting the categories through which the institution attempts to organise, evaluate and guide a messier and more ad hoc reality) they are particular useful in this setting because they help us think about what conditions are necessary for each stage of this process to work. What do academics need to do this? What does success look like? What are the most likely failure modes for the activity? The stages of the model all require a level of TPK which diffusion in itself (even if it’s accompanied by technical training about how to use the tool as an individual) can’t meaningfully provide. In another paper they offer some practical categories through which to think about using AI to enable active forms of learning. I used Claude Opus to summarise this bit of their discussion:

AI as a pre-class preparation tool for foundational knowledge — Students use AI to quickly grasp basic material before deeper work, such as summarising an unread paper in ChatGPT to extract key points ahead of a group discussion.
AI as a personalisation engine for diverse learners — Students use AI to adapt content to their own learning style and needs, such as preparing via AI before a flipped class and then organising the output collaboratively on mind-mapping tools, with options like audio-language conversion or transcription for those facing language or accessibility barriers.
AI as a collaborating team member to be critically evaluated — Students treat AI as a project teammate whose contributions must be checked rather than trusted, such as a group task where each person works with ChatGPT, then critiques and fact-checks its output and reflects on the process.
AI as an individual ideation stage feeding into group work — Students use AI privately to generate and sharpen their own ideas before pooling them in a group, such as researching a problem solo with AI assistance and then collaborating with peers on the same theme to completion.

In another paper the same authors offer a framework for thinking about AI literacy in terms of learning objectives for students:

Know and understand the basic functions of AI tools to support learning
Applying AI knowledge, concepts and applications to support learning
Evaluating AI-generated content enabling higher-order thinking skills development
Comprehending the moral and ethical consequences of AI and making informed decisions regarding its use in various contexts

They suggest a range of teaching and learning activities which could be aligned with each of these ILOs. I thought it was interesting that they advocate a bottom up rather than top-down approach to AI-integration. This is how they describe it on pg 8:

By bottom -up approach, we mean module organisers are at the forefront of driving AI adoption and integration. A top-down strategy, driven by the programme director or education committee, can often be bureaucratic, imposing AI adoption on educators regardless of their readiness, which can lead to resistance, particularly at a time when there is a critical need for staff to enhance their skills to prepare students for the workforce. Further, the top-down approach can often involve a lengthy approval process, requiring more time to implement changes. On the other hand, a bottom-up approach places power with the module organisers and tutors, who champion AI in their teaching and to their peers. The strategy behind this approach is to build support from educators who already integrated or are planning to integrate AI into their teaching practices. This is particularly relevant when integrating disruptive technologies like AI into educational settings.

This fits my intuitions perfectly about the need for academics to willingly engage in this process if it’s going to be real purposeful integration. If it’s done as a triaging response to institutional demands (i.e. rushed through to satisfy the request) it’s not going to be constructively aligned in the deeper meaningful sense discussed earlier in this post. They suggest this module-based bottom up approach needs to be integrated at the programme level which highlights the weakness with the top-down/bottom-up dichotomy: what matters is how AI-integration is organised into relational communities, professional teams who deliberate about more or less shared norms, which is something that ‘top-down’ demand and ‘bottom-up’ creative freedom both tend to obviate. It needs to be willing and coherent. So as well as constructive alignment for each unit, there needs to be alignment for the programme as a whole. This also suggests that AI-integration is something which should be pursued at the programme level, reducing the pressure on each unit.

My suggestion would be that unit leads who want to AI-integrate get the support to do so, on the basis that doing it properly on a smaller scale is better than doing it badly on a larger scale. Then as that process proceeds there’s a need to map the pattern of integration to make these opportunities available to a greater pool of students, so it’s a gradual process of steering integration driven by willing professional communities rather than demanding it. The assumption would be that organisational capability increases on the ground as a result of AI-integration going well: people teach with each other, students take multiple units, learning design teams develop playbooks that work in context. The assumption would be that full AI-integration neither could nor should touch every unit (and that there’s a countervailing rationale for low tech and no tech pedagogy as AI-integration proceed) but that we work towards increasing its range and scope over time by identifying the academics who want to do it, because it fits with the professional choices they’re making about what to teach their students, why and how.

Rather than seeing AI-integration as driven by centralised diffusion (with associated mandates to take advantage of the resulting opportunities) it sees it as a process of targeted cultivation, intending to find ways to spread the seeds of that cultivation in ways which will hopefully take root elsewhere, even on a small scale. It needs to be supplemented by a process of organisational learning which respects the collegiality of teaching teams and tries to deepen it as they react to the facts of AI diffusion on the ground, rather than demanding they change everything they do to meet a centrally imposed model. Integration should have a disciplinary rationale or it shouldn’t be pursued. If it doesn’t then we are effectively substituting academically-defined ILOs for centrally imposed ones that are defined on the basis of (sometimes dubious) vocational imperatives rather than a holistic picture of disciplinary knowledge. It also undermines constructive alignment because if the ILOs don’t match teaching activities and assessment then they inevitably need to change to ensure the coherence of teaching and learning. If we dispense with disciplinary rationale then we are beginning to talk about a very different model of teaching which I don’t think is appropriate for research-intensive universities. If we abandon that principle then we need to be realistic about where it leads, because it ultimately points towards disempowering academics as teachers and degrading research-led teaching as a principle of teaching and learning. A centrally-imposed discipline-blind integration model is a threat to research-led teaching itself, just as a failure to integration is a threat to learning itself because it leaves cognitive outsourcing unaddressed.

#AI #assessment #digitalChange #higherEducation #integration #LLMs

#ai #assessment #digitalchange #highereducation #integration #llms

Gary Ackerman @[email protected] · 2026-05-21 · 15:22 UTC

Grading on a bell curve? There are several reasons to do so, but none are grounded in learning. #eduction #highschool #highereducation #Assessment

#eduction #highschool #highereducation #assessment

Nicola Romanò @[email protected] · 2026-05-21 · 09:51 UTC

In the past couple of years I have been in a lot of meetings centered about the topic of "OMG students are using genAI in assessments what do we doooo?"

After marking a lot of assessments of different type from different courses in different years of study and two different undergraduate programmes, here are my conclusions, some of which I have no way of proving, I know, but that's fine.

1. A lot of students use genAI. In many cases I cannot prove that, it's just a feeling (nobody writes like this, especially not Year 1 non-native speakers), but I had students telling me directly and I do believe them.

2. Looking at marks in the cases above where I have that feeling, I see a wide spread from fails to high A.

3. Following from 2, overall marks have not changed significantly and systematically in any of our courses from the past 5 years. There are of course year-to-year fluctuations, but that's cohort-dependent there seems to be no overall trend.

4. The conclusion you might draw from 2 is that we're rewarding students who are good at using genAI. Possibly, and I have not made up my mind up about this. My answer is that we should change our way of assessing and teaching, rather than trying to "catch" students using genAI. We're currently redesigning our Programmes and that's what we're trying to do. I am in the process of designing a new course and I have tried to do that assuming students will use genAI, but making it so that using it will not be an advantage, and might actually make it more cumbersome to do the assessments.

5. We looked at final year dissertations and plotted marks against %AI writing detected by our submission system. Taking this with a pinch of salt, given that AI detectors are biased and unreliable, there is only a small negative correlation but it's such a small effect size as to be essentially negligible. Again, the distribution of marks is the same as in previous years.

So in conclusion, just like anything else AI related, there's a lot of hype on how this is disruptive and it's terrible or game-changing depending on which side you're on. And yet, in practice...

Has anyone had similar observations? I'd love to hear your thoughts.

#genAI #teaching #highered #undergrad #assessment

#genai #teaching #highered #undergrad #assessment