Cloud Application Integration, Cloud Data Integration Taskflow are unavailable in IICS NA East 2 Services. MDM SaaS Jobs are also impacted due to this.
Incident
Report for Informatica Intelligent Cloud Services
Postmortem
[Outage][IICS NA East 2 Services][Cloud API Manager][Cloud Application Integration][Cloud Data Integration][MDM Customer 360 SaaS][MDM Multidomain SaaS][MDM Product 360 SaaS][MDM Reference 360 SaaS][MDM Supplier 360 SaaS][API Center] INCIDENT START/END: 07-Jan-2025 11:14 to 07-Jan-2025 14:40 UTC
Please find the RCA link available here: https://knowledge.informatica.com/s/article/RCA-for-Incident-on-IICS-NA-East-2-Services-on-Jan-07
Resolved
This incident has been resolved.
Update
The Taskflow runtime and design time issues has been resolved now and services are currently operating normally. We're actively monitoring the services to ensure continued smooth operation.
Update
To resolve the Taskflow issues and switch back to the primary database, the Taskflow services will be down for approximately 30 minutes. We will keep the status page updated as we make progress.
All other services are operating normally.
Next update is at 2.30 PM PT
All other services are operating normally.
Next update is at 2.30 PM PT
Update
We continue to work on the issue and still see performance degradations impacting both runtime and design time activities of taskflow services in NA East 2 POD.
All other services are operating normally.
Next update is at 1.45 PM PT
All other services are operating normally.
Next update is at 1.45 PM PT
Update
We continue to work on the issue actively.
We understand the impact this had on your business processes and wanted to provide a quick summary of the resolution efforts:
As part of the issue resolution, taskflow service was upscaled and the database connectivity was routed to a DR Region.
We identified a performance issue with the underlying Taskflow Database in the DR Region and it is upsized to temporarily address the issue.
Please note that since the DR Database is hosted at a different location, the connectivity is experiencing latency issues, leading to performance degradation.
Currently, we are working to restore the full database service on the main site.
We will keep the status page updated as we make progress.
All other services are operating normally.
Next update is at 1 PM PT
We understand the impact this had on your business processes and wanted to provide a quick summary of the resolution efforts:
As part of the issue resolution, taskflow service was upscaled and the database connectivity was routed to a DR Region.
We identified a performance issue with the underlying Taskflow Database in the DR Region and it is upsized to temporarily address the issue.
Please note that since the DR Database is hosted at a different location, the connectivity is experiencing latency issues, leading to performance degradation.
Currently, we are working to restore the full database service on the main site.
We will keep the status page updated as we make progress.
All other services are operating normally.
Next update is at 1 PM PT
Update
We are still working on this issue on priority with regard to performance degradation affecting the Taskflow service on the NA East 2 pod.
As of now both runtime and design-time performance of the Taskflow services are impacted.
Next update by 1 PM PT
As of now both runtime and design-time performance of the Taskflow services are impacted.
Next update by 1 PM PT
Update
We continue to investigate the issue with regard to performance degradation affecting the Taskflow service on the NA East 2 pod.
Both runtime and design-time functionalities of the Taskflow services are impacted.
Next update by 11:30 am PT
Both runtime and design-time functionalities of the Taskflow services are impacted.
Next update by 11:30 am PT
Update
We have identified intermittent issues resulting from performance degradation affecting the Taskflow service on the NA East 2 pod. Both runtime and design-time functionalities of the Taskflow services are impacted. We are addressing this matter with utmost priority and will provide updates on the status page with further details.
Next update by 10:30 am PT
Next update by 10:30 am PT
Update
We are noticing intermittent issues due to performance degradation for the Taskflow runtime. We are working with high priority. We will update the status page with more details.
Update
We are continuing to monitor for any further issues.
Monitoring
The Taskflow runtime has been resolved. We are validating the design time of the task flow.
We will update the status page once the validations are completed.
We will update the status page once the validations are completed.
Update
Cloud Application Integration services are now available, and executions should resume normalcy. We will update you shortly on the status of other services.
Update
We are continuing to investigate this issue.
Update
We are continuing to investigate this issue.
Update
We are continuing to investigate this issue.
Investigating
Cloud Application Integration and Data Integration Taskflow service and MDM SaaS is unavailable on IICS NA East 2 Services
Both runtime and designtime are impacted for the above services. The API Manager / API Center APIs consuming the Cloud Application Integration process will also be impacted.
We are actively working on this issue. We shall keep the status page updated.
Both runtime and designtime are impacted for the above services. The API Manager / API Center APIs consuming the Cloud Application Integration process will also be impacted.
We are actively working on this issue. We shall keep the status page updated.
This incident affected: IICS NA East 2 Services (Cloud API Manager, Cloud Application Integration, Cloud Data Integration, MDM Customer 360 SaaS, MDM Multidomain SaaS, MDM Product 360 SaaS, MDM Reference 360 SaaS, MDM Supplier 360 SaaS, API Center).