1
// This file is part of Substrate.
2

            
3
// Copyright (C) Parity Technologies (UK) Ltd.
4
// SPDX-License-Identifier: Apache-2.0
5

            
6
// Licensed under the Apache License, Version 2.0 (the "License");
7
// you may not use this file except in compliance with the License.
8
// You may obtain a copy of the License at
9
//
10
// 	http://www.apache.org/licenses/LICENSE-2.0
11
//
12
// Unless required by applicable law or agreed to in writing, software
13
// distributed under the License is distributed on an "AS IS" BASIS,
14
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15
// See the License for the specific language governing permissions and
16
// limitations under the License.
17

            
18
#![deny(missing_docs)]
19
#![deny(rustdoc::broken_intra_doc_links)]
20

            
21
//! # `pallet-migrations`
22
//!
23
//! Provides multi block migrations for FRAME runtimes.
24
//!
25
//! ## Overview
26
//!
27
//! The pallet takes care of executing a batch of multi-step migrations over multiple blocks. The
28
//! process starts on each runtime upgrade. Normal and operational transactions are paused while
29
//! migrations are on-going.
30
//!
31
//! ### Example
32
//!
33
//! This example demonstrates a simple mocked walk through of a basic success scenario. The pallet
34
//! is configured with two migrations: one succeeding after just one step, and the second one
35
//! succeeding after two steps. A runtime upgrade is then enacted and the block number is advanced
36
//! until all migrations finish executing. Afterwards, the recorded historic migrations are
37
//! checked and events are asserted.
38
#![doc = docify::embed!("src/tests.rs", simple_works)]
39
//!
40
//! ## Pallet API
41
//!
42
//! See the [`pallet`] module for more information about the interfaces this pallet exposes,
43
//! including its configuration trait, dispatchables, storage items, events and errors.
44
//!
45
//! Otherwise noteworthy API of this pallet include its implementation of the
46
//! [`MultiStepMigrator`] trait. This must be plugged into
47
//! [`frame_system::Config::MultiBlockMigrator`] for proper function.
48
//!
49
//! The API contains some calls for emergency management. They are all prefixed with `force_` and
50
//! should normally not be needed. Pay special attention prior to using them.
51
//!
52
//! ### Design Goals
53
//!
54
//! 1. Must automatically execute migrations over multiple blocks.
55
//! 2. Must expose information about whether migrations are ongoing.
56
//! 3. Must respect pessimistic weight bounds of migrations.
57
//! 4. Must execute migrations in order. Skipping is not allowed; migrations are run on a
58
//! all-or-nothing basis.
59
//! 5. Must prevent re-execution of past migrations.
60
//! 6. Must provide transactional storage semantics for migrations.
61
//! 7. Must guarantee progress.
62
//!
63
//! ### Design
64
//!
65
//! Migrations are provided to the pallet through the associated type [`Config::Migrations`] of type
66
//! [`SteppedMigrations`]. This allows multiple migrations to be aggregated through a tuple. It
67
//! simplifies the trait bounds since all associated types of the trait must be provided by the
68
//! pallet. The actual progress of the pallet is stored in the [`Cursor`] storage item. This can
69
//! either be [`MigrationCursor::Active`] or [`MigrationCursor::Stuck`]. In the active case it
70
//! points to the currently active migration and stores its inner cursor. The inner cursor can then
71
//! be used by the migration to store its inner state and advance. Each time when the migration
72
//! returns `Some(cursor)`, it signals the pallet that it is not done yet.
73
//! The cursor is reset on each runtime upgrade. This ensures that it starts to execute at the
74
//! first migration in the vector. The pallets cursor is only ever incremented or set to `Stuck`
75
//! once it encounters an error (Goal 4). Once in the stuck state, the pallet will stay stuck until
76
//! it is fixed through manual governance intervention.
77
//! As soon as the cursor of the pallet becomes `Some(_)`; [`MultiStepMigrator::ongoing`] returns
78
//! `true` (Goal 2). This can be used by upstream code to possibly pause transactions.
79
//! In `on_initialize` the pallet will load the current migration and check whether it was already
80
//! executed in the past by checking for membership of its ID in the [`Historic`] set. Historic
81
//! migrations are skipped without causing an error. Each successfully executed migration is added
82
//! to this set (Goal 5).
83
//! This proceeds until no more migrations remain. At that point, the event `UpgradeCompleted` is
84
//! emitted (Goal 1).
85
//! The execution of each migration happens by calling [`SteppedMigration::transactional_step`].
86
//! This function wraps the inner `step` function into a transactional layer to allow rollback in
87
//! the error case (Goal 6).
88
//! Weight limits must be checked by the migration itself. The pallet provides a [`WeightMeter`] for
89
//! that purpose. The pallet may return [`SteppedMigrationError::InsufficientWeight`] at any point.
90
//! In that scenario, one of two things will happen: if that migration was exclusively executed
91
//! in this block, and therefore required more than the maximum amount of weight possible, the
92
//! process becomes `Stuck`. Otherwise, one re-attempt is executed with the same logic in the next
93
//! block (Goal 3). Progress through the migrations is guaranteed by providing a timeout for each
94
//! migration via [`SteppedMigration::max_steps`]. The pallet **ONLY** guarantees progress if this
95
//! is set to sensible limits (Goal 7).
96
//!
97
//! ### Scenario: Governance cleanup
98
//!
99
//! Every now and then, governance can make use of the [`clear_historic`][Pallet::clear_historic]
100
//! call. This ensures that no old migrations pile up in the [`Historic`] set. This can be done very
101
//! rarely, since the storage should not grow quickly and the lookup weight does not suffer much.
102
//! Another possibility would be to have a synchronous single-block migration perpetually deployed
103
//! that cleans them up before the MBMs start.
104
//!
105
//! ### Scenario: Successful upgrade
106
//!
107
//! The standard procedure for a successful runtime upgrade can look like this:
108
//! 1. Migrations are configured in the `Migrations` config item. All migrations expose
109
//! [`max_steps`][SteppedMigration::max_steps], are error tolerant, check their weight bounds and
110
//! have a unique identifier.
111
//! 2. The runtime upgrade is enacted. An `UpgradeStarted` event is
112
//! followed by lots of `MigrationAdvanced` and `MigrationCompleted` events. Finally
113
//! `UpgradeCompleted` is emitted.
114
//! 3. Cleanup as described in the governance scenario be executed at any time after the migrations
115
//! completed.
116
//!
117
//! ### Advice: Failed upgrades
118
//!
119
//! Failed upgrades cannot be recovered from automatically and require governance intervention. Set
120
//! up monitoring for `UpgradeFailed` events to be made aware of any failures. The hook
121
//! [`FailedMigrationHandler::failed`] should be setup in a way that it allows governance to act,
122
//! but still prevent other transactions from interacting with the inconsistent storage state. Note
123
//! that this is paramount, since the inconsistent state might contain a faulty balance amount or
124
//! similar that could cause great harm if user transactions don't remain suspended. One way to
125
//! implement this would be to use the `SafeMode` or `TxPause` pallets that can prevent most user
126
//! interactions but still allow a whitelisted set of governance calls.
127
//!
128
//! ### Remark: Failed migrations
129
//!
130
//! Failed migrations are not added to the `Historic` set. This means that an erroneous
131
//! migration must be removed and fixed manually. This already applies, even before considering the
132
//! historic set.
133
//!
134
//! ### Remark: Transactional processing
135
//!
136
//! You can see the transactional semantics for migration steps as mostly useless, since in the
137
//! stuck case the state is already messed up. This just prevents it from becoming even more messed
138
//! up, but doesn't prevent it in the first place.
139

            
140
#![cfg_attr(not(feature = "std"), no_std)]
141

            
142
mod benchmarking;
143
mod mock;
144
pub mod mock_helpers;
145
mod tests;
146
pub mod weights;
147

            
148
extern crate alloc;
149

            
150
pub use pallet::*;
151
pub use weights::WeightInfo;
152

            
153
use alloc::vec::Vec;
154
use codec::{Decode, Encode, MaxEncodedLen};
155
use core::ops::ControlFlow;
156
use frame_support::{
157
	defensive, defensive_assert,
158
	migrations::*,
159
	traits::Get,
160
	weights::{Weight, WeightMeter},
161
	BoundedVec,
162
};
163
use frame_system::{pallet_prelude::BlockNumberFor, Pallet as System};
164
use sp_runtime::Saturating;
165

            
166
/// Points to the next migration to execute.
167
18
#[derive(Debug, Clone, Eq, PartialEq, Encode, Decode, scale_info::TypeInfo, MaxEncodedLen)]
168
pub enum MigrationCursor<Cursor, BlockNumber> {
169
669
	/// Points to the currently active migration and its inner cursor.
170
669
	Active(ActiveCursor<Cursor, BlockNumber>),
171

            
172
141
	/// Migration got stuck and cannot proceed. This is bad.
173
141
	Stuck,
174
}
175

            
176
impl<Cursor, BlockNumber> MigrationCursor<Cursor, BlockNumber> {
177
	/// Try to return self as an [`ActiveCursor`].
178
	pub fn as_active(&self) -> Option<&ActiveCursor<Cursor, BlockNumber>> {
179
		match self {
180
			MigrationCursor::Active(active) => Some(active),
181
			MigrationCursor::Stuck => None,
182
		}
183
	}
184
}
185

            
186
impl<Cursor, BlockNumber> From<ActiveCursor<Cursor, BlockNumber>>
187
	for MigrationCursor<Cursor, BlockNumber>
188
{
189
	fn from(active: ActiveCursor<Cursor, BlockNumber>) -> Self {
190
		MigrationCursor::Active(active)
191
	}
192
}
193

            
194
/// Points to the currently active migration and its inner cursor.
195
#[derive(Debug, Clone, Eq, PartialEq, Encode, Decode, scale_info::TypeInfo, MaxEncodedLen)]
196
pub struct ActiveCursor<Cursor, BlockNumber> {
197
	/// The index of the migration in the MBM tuple.
198
	pub index: u32,
199
	/// The cursor of the migration that is referenced by `index`.
200
	pub inner_cursor: Option<Cursor>,
201
	/// The block number that the migration started at.
202
	///
203
	/// This is used to calculate how many blocks it took.
204
	pub started_at: BlockNumber,
205
}
206

            
207
impl<Cursor, BlockNumber> ActiveCursor<Cursor, BlockNumber> {
208
	/// Advance the overarching cursor to the next migration.
209
	pub(crate) fn goto_next_migration(&mut self, current_block: BlockNumber) {
210
		self.index.saturating_inc();
211
		self.inner_cursor = None;
212
		self.started_at = current_block;
213
	}
214
}
215

            
216
/// How to clear the records of historic migrations.
217
9
#[derive(Debug, Clone, Eq, PartialEq, Encode, Decode, scale_info::TypeInfo)]
218
pub enum HistoricCleanupSelector<Id> {
219
402
	/// Clear exactly these entries.
220
	///
221
	/// This is the advised way of doing it.
222
402
	Specific(Vec<Id>),
223

            
224
228
	/// Clear up to this many entries
225
	Wildcard {
226
6
		/// How many should be cleared in this call at most.
227
6
		limit: Option<u32>,
228
3
		/// The cursor that was emitted from any previous `HistoricCleared`.
229
3
		///
230
3
		/// Does not need to be passed when clearing the first batch.
231
3
		previous_cursor: Option<Vec<u8>>,
232
228
	},
233
}
234

            
235
/// The default number of entries that should be cleared by a `HistoricCleanupSelector::Wildcard`.
236
///
237
/// The caller can explicitly specify a higher amount. Benchmarks are run with twice this value.
238
const DEFAULT_HISTORIC_BATCH_CLEAR_SIZE: u32 = 128;
239

            
240
impl<Id> HistoricCleanupSelector<Id> {
241
	/// The maximal number of entries that this will remove.
242
	///
243
	/// Needed for weight calculation.
244
459
	pub fn limit(&self) -> u32 {
245
459
		match self {
246
			Self::Specific(ids) => ids.len() as u32,
247
186
			Self::Wildcard { limit, .. } => limit.unwrap_or(DEFAULT_HISTORIC_BATCH_CLEAR_SIZE),
248
		}
249
459
	}
250
}
251

            
252
/// Convenience alias for [`MigrationCursor`].
253
pub type CursorOf<T> = MigrationCursor<RawCursorOf<T>, BlockNumberFor<T>>;
254

            
255
/// Convenience alias for the raw inner cursor of a migration.
256
pub type RawCursorOf<T> = BoundedVec<u8, <T as Config>::CursorMaxLen>;
257

            
258
/// Convenience alias for the identifier of a migration.
259
pub type IdentifierOf<T> = BoundedVec<u8, <T as Config>::IdentifierMaxLen>;
260

            
261
/// Convenience alias for [`ActiveCursor`].
262
pub type ActiveCursorOf<T> = ActiveCursor<RawCursorOf<T>, BlockNumberFor<T>>;
263

            
264
/// Trait for a tuple of No-OP migrations with one element.
265
pub trait MockedMigrations: SteppedMigrations {
266
	/// The migration should fail after `n` steps.
267
	fn set_fail_after(n: u32);
268
	/// The migration should succeed after `n` steps.
269
	fn set_success_after(n: u32);
270
}
271

            
272
765
#[frame_support::pallet]
273
pub mod pallet {
274
	use super::*;
275
	use frame_support::pallet_prelude::*;
276
	use frame_system::pallet_prelude::*;
277

            
278
1698
	#[pallet::pallet]
279
	pub struct Pallet<T>(_);
280

            
281
	#[pallet::config(with_default)]
282
	pub trait Config: frame_system::Config {
283
		/// The overarching event type of the runtime.
284
		#[pallet::no_default_bounds]
285
		type RuntimeEvent: From<Event<Self>> + IsType<<Self as frame_system::Config>::RuntimeEvent>;
286

            
287
		/// All the multi-block migrations to run.
288
		///
289
		/// Should only be updated in a runtime-upgrade once all the old migrations have completed.
290
		/// (Check that [`Cursor`] is `None`).
291
		#[cfg(not(feature = "runtime-benchmarks"))]
292
		#[pallet::no_default]
293
		type Migrations: SteppedMigrations;
294

            
295
		/// Mocked migrations for benchmarking only.
296
		///
297
		/// Should be configured to [`crate::mock_helpers::MockedMigrations`] in benchmarks.
298
		#[cfg(feature = "runtime-benchmarks")]
299
		#[pallet::no_default]
300
		type Migrations: MockedMigrations;
301

            
302
		/// The maximal length of an encoded cursor.
303
		///
304
		/// A good default needs to selected such that no migration will ever have a cursor with MEL
305
		/// above this limit. This is statically checked in `integrity_test`.
306
		#[pallet::constant]
307
		type CursorMaxLen: Get<u32>;
308

            
309
		/// The maximal length of an encoded identifier.
310
		///
311
		/// A good default needs to selected such that no migration will ever have an identifier
312
		/// with MEL above this limit. This is statically checked in `integrity_test`.
313
		#[pallet::constant]
314
		type IdentifierMaxLen: Get<u32>;
315

            
316
		/// Notifications for status updates of a runtime upgrade.
317
		///
318
		/// Could be used to pause XCM etc.
319
		type MigrationStatusHandler: MigrationStatusHandler;
320

            
321
		/// Handler for failed migrations.
322
		type FailedMigrationHandler: FailedMigrationHandler;
323

            
324
		/// The maximum weight to spend each block to execute migrations.
325
		type MaxServiceWeight: Get<Weight>;
326

            
327
		/// Weight information for the calls and functions of this pallet.
328
		type WeightInfo: WeightInfo;
329
	}
330

            
331
	/// Default implementations of [`DefaultConfig`], which can be used to implement [`Config`].
332
	pub mod config_preludes {
333
		use super::{inject_runtime_type, DefaultConfig};
334
		use frame_support::{
335
			derive_impl,
336
			migrations::FreezeChainOnFailedMigration,
337
			pallet_prelude::{ConstU32, *},
338
		};
339
		use frame_system::limits::BlockWeights;
340

            
341
		/// Provides a viable default config that can be used with
342
		/// [`derive_impl`](`frame_support::derive_impl`) to derive a testing pallet config
343
		/// based on this one.
344
		///
345
		/// See `Test` in the `default-config` example pallet's `test.rs` for an example of
346
		/// a downstream user of this particular `TestDefaultConfig`
347
		pub struct TestDefaultConfig;
348

            
349
		frame_support::parameter_types! {
350
			/// Maximal weight per block that can be spent on migrations in tests.
351
			pub TestMaxServiceWeight: Weight = <<TestDefaultConfig as frame_system::DefaultConfig>::BlockWeights as Get<BlockWeights>>::get().max_block.div(2);
352
		}
353

            
354
		#[derive_impl(frame_system::config_preludes::TestDefaultConfig, no_aggregated_types)]
355
		impl frame_system::DefaultConfig for TestDefaultConfig {}
356

            
357
		#[frame_support::register_default_impl(TestDefaultConfig)]
358
		impl DefaultConfig for TestDefaultConfig {
359
			#[inject_runtime_type]
360
			type RuntimeEvent = ();
361
			type CursorMaxLen = ConstU32<{ 1 << 16 }>;
362
			type IdentifierMaxLen = ConstU32<{ 256 }>;
363
			type MigrationStatusHandler = ();
364
			type FailedMigrationHandler = FreezeChainOnFailedMigration;
365
			type MaxServiceWeight = TestMaxServiceWeight;
366
			type WeightInfo = ();
367
		}
368
	}
369

            
370
	/// The currently active migration to run and its cursor.
371
	///
372
	/// `None` indicates that no migration is running.
373
2094474
	#[pallet::storage]
374
	pub type Cursor<T: Config> = StorageValue<_, CursorOf<T>, OptionQuery>;
375

            
376
	/// Set of all successfully executed migrations.
377
	///
378
	/// This is used as blacklist, to not re-execute migrations that have not been removed from the
379
	/// codebase yet. Governance can regularly clear this out via `clear_historic`.
380
	#[pallet::storage]
381
	pub type Historic<T: Config> = StorageMap<_, Twox64Concat, IdentifierOf<T>, (), OptionQuery>;
382

            
383
	#[pallet::event]
384
	#[pallet::generate_deposit(pub(super) fn deposit_event)]
385
	pub enum Event<T: Config> {
386
		/// A Runtime upgrade started.
387
		///
388
		/// Its end is indicated by `UpgradeCompleted` or `UpgradeFailed`.
389
		UpgradeStarted {
390
			/// The number of migrations that this upgrade contains.
391
			///
392
			/// This can be used to design a progress indicator in combination with counting the
393
			/// `MigrationCompleted` and `MigrationSkipped` events.
394
			migrations: u32,
395
		},
396
		/// The current runtime upgrade completed.
397
		///
398
		/// This implies that all of its migrations completed successfully as well.
399
		UpgradeCompleted,
400
		/// Runtime upgrade failed.
401
		///
402
		/// This is very bad and will require governance intervention.
403
		UpgradeFailed,
404
		/// A migration was skipped since it was already executed in the past.
405
		MigrationSkipped {
406
			/// The index of the skipped migration within the [`Config::Migrations`] list.
407
			index: u32,
408
		},
409
		/// A migration progressed.
410
		MigrationAdvanced {
411
			/// The index of the migration within the [`Config::Migrations`] list.
412
			index: u32,
413
			/// The number of blocks that this migration took so far.
414
			took: BlockNumberFor<T>,
415
		},
416
		/// A Migration completed.
417
		MigrationCompleted {
418
			/// The index of the migration within the [`Config::Migrations`] list.
419
			index: u32,
420
			/// The number of blocks that this migration took so far.
421
			took: BlockNumberFor<T>,
422
		},
423
		/// A Migration failed.
424
		///
425
		/// This implies that the whole upgrade failed and governance intervention is required.
426
		MigrationFailed {
427
			/// The index of the migration within the [`Config::Migrations`] list.
428
			index: u32,
429
			/// The number of blocks that this migration took so far.
430
			took: BlockNumberFor<T>,
431
		},
432
		/// The set of historical migrations has been cleared.
433
		HistoricCleared {
434
			/// Should be passed to `clear_historic` in a successive call.
435
			next_cursor: Option<Vec<u8>>,
436
		},
437
	}
438

            
439
	#[pallet::error]
440
	pub enum Error<T> {
441
		/// The operation cannot complete since some MBMs are ongoing.
442
		Ongoing,
443
	}
444

            
445
554367
	#[pallet::hooks]
446
	impl<T: Config> Hooks<BlockNumberFor<T>> for Pallet<T> {
447
		fn on_runtime_upgrade() -> Weight {
448
			Self::onboard_new_mbms()
449
		}
450

            
451
		#[cfg(feature = "std")]
452
53637
		fn integrity_test() {
453
53637
			// Check that the migrations tuple is legit.
454
53637
			frame_support::assert_ok!(T::Migrations::integrity_test());
455

            
456
			// Very important! Ensure that the pallet is configured in `System::Config`.
457
			{
458
53637
				assert!(!Cursor::<T>::exists(), "Externalities storage should be clean");
459
53637
				assert!(!<T as frame_system::Config>::MultiBlockMigrator::ongoing());
460

            
461
53637
				Cursor::<T>::put(MigrationCursor::Stuck);
462
53637
				assert!(<T as frame_system::Config>::MultiBlockMigrator::ongoing());
463

            
464
53637
				Cursor::<T>::kill();
465
53637
			}
466
53637

            
467
53637
			// The per-block service weight is sane.
468
53637
			{
469
53637
				let want = T::MaxServiceWeight::get();
470
53637
				let max = <T as frame_system::Config>::BlockWeights::get().max_block;
471
53637

            
472
53637
				assert!(want.all_lte(max), "Service weight is larger than a block: {want} > {max}");
473
			}
474

            
475
			// Cursor MEL
476
			{
477
53637
				let mel = T::Migrations::cursor_max_encoded_len();
478
53637
				let max_mel = T::CursorMaxLen::get() as usize;
479
53637
				assert!(
480
53637
					mel <= max_mel,
481
					"A Cursor is not guaranteed to fit into the storage: {mel} > {max_mel}",
482
				);
483
			}
484

            
485
			// Identifier MEL
486
			{
487
53637
				let mel = T::Migrations::identifier_max_encoded_len();
488
53637
				let max_mel = T::IdentifierMaxLen::get() as usize;
489
53637
				assert!(
490
53637
					mel <= max_mel,
491
					"An Identifier is not guaranteed to fit into the storage: {mel} > {max_mel}",
492
				);
493
			}
494
53637
		}
495
	}
496

            
497
5730
	#[pallet::call(weight = T::WeightInfo)]
498
	impl<T: Config> Pallet<T> {
499
		/// Allows root to set a cursor to forcefully start, stop or forward the migration process.
500
		///
501
		/// Should normally not be needed and is only in place as emergency measure. Note that
502
		/// restarting the migration process in this manner will not call the
503
		/// [`MigrationStatusHandler::started`] hook or emit an `UpgradeStarted` event.
504
		#[pallet::call_index(0)]
505
		pub fn force_set_cursor(
506
			origin: OriginFor<T>,
507
			cursor: Option<CursorOf<T>>,
508
270
		) -> DispatchResult {
509
270
			ensure_root(origin)?;
510

            
511
			Cursor::<T>::set(cursor);
512

            
513
			Ok(())
514
		}
515

            
516
		/// Allows root to set an active cursor to forcefully start/forward the migration process.
517
		///
518
		/// This is an edge-case version of [`Self::force_set_cursor`] that allows to set the
519
		/// `started_at` value to the next block number. Otherwise this would not be possible, since
520
		/// `force_set_cursor` takes an absolute block number. Setting `started_at` to `None`
521
		/// indicates that the current block number plus one should be used.
522
		#[pallet::call_index(1)]
523
		pub fn force_set_active_cursor(
524
			origin: OriginFor<T>,
525
			index: u32,
526
			inner_cursor: Option<RawCursorOf<T>>,
527
			started_at: Option<BlockNumberFor<T>>,
528
231
		) -> DispatchResult {
529
231
			ensure_root(origin)?;
530

            
531
			let started_at = started_at.unwrap_or(
532
				System::<T>::block_number().saturating_add(sp_runtime::traits::One::one()),
533
			);
534
			Cursor::<T>::put(MigrationCursor::Active(ActiveCursor {
535
				index,
536
				inner_cursor,
537
				started_at,
538
			}));
539

            
540
			Ok(())
541
		}
542

            
543
		/// Forces the onboarding of the migrations.
544
		///
545
		/// This process happens automatically on a runtime upgrade. It is in place as an emergency
546
		/// measurement. The cursor needs to be `None` for this to succeed.
547
		#[pallet::call_index(2)]
548
57
		pub fn force_onboard_mbms(origin: OriginFor<T>) -> DispatchResult {
549
57
			ensure_root(origin)?;
550

            
551
			ensure!(!Cursor::<T>::exists(), Error::<T>::Ongoing);
552
			Self::onboard_new_mbms();
553

            
554
			Ok(())
555
		}
556

            
557
		/// Clears the `Historic` set.
558
		///
559
		/// `map_cursor` must be set to the last value that was returned by the
560
		/// `HistoricCleared` event. The first time `None` can be used. `limit` must be chosen in a
561
		/// way that will result in a sensible weight.
562
		#[pallet::call_index(3)]
563
		#[pallet::weight(T::WeightInfo::clear_historic(selector.limit()))]
564
		pub fn clear_historic(
565
			origin: OriginFor<T>,
566
			selector: HistoricCleanupSelector<IdentifierOf<T>>,
567
204
		) -> DispatchResult {
568
204
			ensure_root(origin)?;
569

            
570
			match &selector {
571
				HistoricCleanupSelector::Specific(ids) => {
572
					for id in ids {
573
						Historic::<T>::remove(id);
574
					}
575
					Self::deposit_event(Event::HistoricCleared { next_cursor: None });
576
				},
577
				HistoricCleanupSelector::Wildcard { previous_cursor, .. } => {
578
					let next = Historic::<T>::clear(selector.limit(), previous_cursor.as_deref());
579
					Self::deposit_event(Event::HistoricCleared { next_cursor: next.maybe_cursor });
580
				},
581
			}
582

            
583
			Ok(())
584
		}
585
	}
586
}
587

            
588
impl<T: Config> Pallet<T> {
589
	/// Onboard all new Multi-Block-Migrations and start the process of executing them.
590
	///
591
	/// Should only be called once all previous migrations completed.
592
	fn onboard_new_mbms() -> Weight {
593
		if let Some(cursor) = Cursor::<T>::get() {
594
			log::error!("Ongoing migrations interrupted - chain stuck");
595

            
596
			let maybe_index = cursor.as_active().map(|c| c.index);
597
			Self::upgrade_failed(maybe_index);
598
			return T::WeightInfo::onboard_new_mbms()
599
		}
600

            
601
		let migrations = T::Migrations::len();
602
		log::debug!("Onboarding {migrations} new MBM migrations");
603

            
604
		if migrations > 0 {
605
			// Set the cursor to the first migration:
606
			Cursor::<T>::set(Some(
607
				ActiveCursor {
608
					index: 0,
609
					inner_cursor: None,
610
					started_at: System::<T>::block_number(),
611
				}
612
				.into(),
613
			));
614
			Self::deposit_event(Event::UpgradeStarted { migrations });
615
			T::MigrationStatusHandler::started();
616
		}
617

            
618
		T::WeightInfo::onboard_new_mbms()
619
	}
620

            
621
	/// Tries to make progress on the Multi-Block-Migrations process.
622
	fn progress_mbms(n: BlockNumberFor<T>) -> Weight {
623
		let mut meter = WeightMeter::with_limit(T::MaxServiceWeight::get());
624
		meter.consume(T::WeightInfo::progress_mbms_none());
625

            
626
		let mut cursor = match Cursor::<T>::get() {
627
			None => {
628
				log::trace!("[Block {n:?}] Waiting for cursor to become `Some`.");
629
				return meter.consumed()
630
			},
631
			Some(MigrationCursor::Active(cursor)) => {
632
				log::debug!("Progressing MBM #{}", cursor.index);
633
				cursor
634
			},
635
			Some(MigrationCursor::Stuck) => {
636
				log::error!("Migration stuck. Governance intervention required.");
637
				return meter.consumed()
638
			},
639
		};
640
		debug_assert!(Self::ongoing());
641

            
642
		// The limit here is a defensive measure to prevent an infinite loop. It expresses that we
643
		// allow no more than 8 MBMs to finish in a single block. This should be harmless, since we
644
		// generally expect *Multi*-Block-Migrations to take *multiple* blocks.
645
		for i in 0..8 {
646
			match Self::exec_migration(cursor, i == 0, &mut meter) {
647
				None => return meter.consumed(),
648
				Some(ControlFlow::Continue(next_cursor)) => {
649
					cursor = next_cursor;
650
				},
651
				Some(ControlFlow::Break(last_cursor)) => {
652
					cursor = last_cursor;
653
					break
654
				},
655
			}
656
		}
657

            
658
		Cursor::<T>::set(Some(cursor.into()));
659

            
660
		meter.consumed()
661
	}
662

            
663
	/// Try to make progress on the current migration.
664
	///
665
	/// Returns whether processing should continue or break for this block. The return value means:
666
	/// - `None`: The migration process is completely finished.
667
	/// - `ControlFlow::Break`: Continue in the *next* block with the given cursor.
668
	/// - `ControlFlow::Continue`: Continue in the *current* block with the given cursor.
669
	fn exec_migration(
670
		mut cursor: ActiveCursorOf<T>,
671
		is_first: bool,
672
		meter: &mut WeightMeter,
673
	) -> Option<ControlFlow<ActiveCursorOf<T>, ActiveCursorOf<T>>> {
674
		// The differences between the single branches' weights is not that big. And since we do
675
		// only one step per block, we can just use the maximum instead of more precise accounting.
676
		if meter.try_consume(Self::exec_migration_max_weight()).is_err() {
677
			defensive_assert!(!is_first, "There should be enough weight to do this at least once");
678
			return Some(ControlFlow::Break(cursor))
679
		}
680

            
681
		let Some(id) = T::Migrations::nth_id(cursor.index) else {
682
			// No more migrations in the tuple - we are done.
683
			defensive_assert!(cursor.index == T::Migrations::len(), "Inconsistent MBMs tuple");
684
			Self::deposit_event(Event::UpgradeCompleted);
685
			Cursor::<T>::kill();
686
			T::MigrationStatusHandler::completed();
687
			return None;
688
		};
689

            
690
		let Ok(bounded_id): Result<IdentifierOf<T>, _> = id.try_into() else {
691
			defensive!("integrity_test ensures that all identifiers' MEL bounds fit into CursorMaxLen; qed.");
692
			Self::upgrade_failed(Some(cursor.index));
693
			return None
694
		};
695

            
696
		if Historic::<T>::contains_key(&bounded_id) {
697
			Self::deposit_event(Event::MigrationSkipped { index: cursor.index });
698
			cursor.goto_next_migration(System::<T>::block_number());
699
			return Some(ControlFlow::Continue(cursor))
700
		}
701

            
702
		let max_steps = T::Migrations::nth_max_steps(cursor.index);
703
		let next_cursor = T::Migrations::nth_transactional_step(
704
			cursor.index,
705
			cursor.inner_cursor.clone().map(|c| c.into_inner()),
706
			meter,
707
		);
708
		let Some((max_steps, next_cursor)) = max_steps.zip(next_cursor) else {
709
			defensive!("integrity_test ensures that the tuple is valid; qed");
710
			Self::upgrade_failed(Some(cursor.index));
711
			return None
712
		};
713

            
714
		let took = System::<T>::block_number().saturating_sub(cursor.started_at);
715
		match next_cursor {
716
			Ok(Some(next_cursor)) => {
717
				let Ok(bound_next_cursor) = next_cursor.try_into() else {
718
					defensive!("The integrity check ensures that all cursors' MEL bound fits into CursorMaxLen; qed");
719
					Self::upgrade_failed(Some(cursor.index));
720
					return None
721
				};
722

            
723
				Self::deposit_event(Event::MigrationAdvanced { index: cursor.index, took });
724
				cursor.inner_cursor = Some(bound_next_cursor);
725

            
726
				if max_steps.map_or(false, |max| took > max.into()) {
727
					Self::deposit_event(Event::MigrationFailed { index: cursor.index, took });
728
					Self::upgrade_failed(Some(cursor.index));
729
					None
730
				} else {
731
					// A migration cannot progress more than one step per block, we therefore break.
732
					Some(ControlFlow::Break(cursor))
733
				}
734
			},
735
			Ok(None) => {
736
				// A migration is done when it returns cursor `None`.
737
				Self::deposit_event(Event::MigrationCompleted { index: cursor.index, took });
738
				Historic::<T>::insert(&bounded_id, ());
739
				cursor.goto_next_migration(System::<T>::block_number());
740
				Some(ControlFlow::Continue(cursor))
741
			},
742
			Err(SteppedMigrationError::InsufficientWeight { required }) => {
743
				if is_first || required.any_gt(meter.limit()) {
744
					Self::deposit_event(Event::MigrationFailed { index: cursor.index, took });
745
					Self::upgrade_failed(Some(cursor.index));
746
					None
747
				} else {
748
					// Retry and hope that there is more weight in the next block.
749
					Some(ControlFlow::Break(cursor))
750
				}
751
			},
752
			Err(SteppedMigrationError::InvalidCursor | SteppedMigrationError::Failed) => {
753
				Self::deposit_event(Event::MigrationFailed { index: cursor.index, took });
754
				Self::upgrade_failed(Some(cursor.index));
755
				None
756
			},
757
		}
758
	}
759

            
760
	/// Fail the current runtime upgrade, caused by `migration`.
761
	fn upgrade_failed(migration: Option<u32>) {
762
		use FailedMigrationHandling::*;
763
		Self::deposit_event(Event::UpgradeFailed);
764

            
765
		match T::FailedMigrationHandler::failed(migration) {
766
			KeepStuck => Cursor::<T>::set(Some(MigrationCursor::Stuck)),
767
			ForceUnstuck => Cursor::<T>::kill(),
768
			Ignore => {},
769
		}
770
	}
771

            
772
	/// The maximal weight of calling the private `Self::exec_migration` function.
773
	pub fn exec_migration_max_weight() -> Weight {
774
		T::WeightInfo::exec_migration_complete()
775
			.max(T::WeightInfo::exec_migration_completed())
776
			.max(T::WeightInfo::exec_migration_skipped_historic())
777
			.max(T::WeightInfo::exec_migration_advance())
778
			.max(T::WeightInfo::exec_migration_fail())
779
	}
780
}
781

            
782
impl<T: Config> MultiStepMigrator for Pallet<T> {
783
886326
	fn ongoing() -> bool {
784
886326
		Cursor::<T>::exists()
785
886326
	}
786

            
787
	fn step() -> Weight {
788
		Self::progress_mbms(System::<T>::block_number())
789
	}
790
}