Subversion Repositories ion

--------------------------------------------------------------------------------

-- mips_cache.vhdl -- cache + memory interface module

--

-- This module contains both MIPS caches (I-Cache and D-Cache) combined with

-- all the glue logic used to decode and interface external memories and

-- devices, both synchronous and asynchronous. 

-- Everything that goes into or comes from the CPU passes through this module.

--

-- See a list of known problems at the bottom of this header.

-- 

--------------------------------------------------------------------------------

-- Main cache parameters:

--

-- I-Cache: 256 4-word lines, direct mapped.

-- D-Cache: 256 4-word lines, direct mapped, write-through

--

-- The cache works mostly like the R3000 caches, except for the following 

-- traits:

--

-- 1.- When bit CP0[12].17='0' (reset value) the cache is 'disabled'. In this 

-- state, ALL memory reads miss the cache and force a line refill -- even 

-- succesive reads from the same line will refill the entire line. This 

-- simplifies the cache logic a lot but slows uncached code a lot. Which means 

-- you should initialize the cache and enable it ASAP after reset. 

-- 

-- 2.- When bits CP0[12].17:16 = "01", the CPU can invalidate a cache line N

-- by writing word N to ANY address. The address will be executed as normal AND

-- the cache controller will invalidate I-Cache line N.

--

-- Note that the standard behavior for bits 17 and 16 of the SR is not

-- implemented at all -- no cache swapping, etc.

--

-- 3.- In this version, all areas of memory are cacheable, except those mapped 

-- as MT_IO_SYNC or MT_UNMAPPED in mips_pkg. 

-- Since you can enable or disable the cache at will this difference doesn't 

-- seem too important.

-- There is a 'cacheable' flag in the t_range_attr record which is currently 

-- unused.

--

-- 4.- The tag is only 14 bits long, which means the memory map is severely

-- restricted in this version. See @note2.

--

-- This is not the standard MIPS way but is compatible enough and above all it

-- is simple.

--

--------------------------------------------------------------------------------

-- NOTES:

--

-- @note1: I-Cache initialization and tag format

--

-- In the tag table (code_tag_table), tags are stored together with a 'valid' 

-- bit (MSB), which is '0' for VALID tags.

-- When the CPU invalidates a line, it writes a '1' in the proper tag table 

-- entry together with the tag value.

-- When tags are matched, the valid bit is matched against 

--

--

-- @note2: I-Cache tags and cache mirroring

-- 

-- To save space in the I-Cache tag table, the tags are shorter than they 

-- should -- 14 bits instead of the 20 bits we would need to cover the

-- entire 32-bit address:

--

--             ___________ <-- These address bits are NOT in the tag

--            /           \

--  31 ..   27| 26 .. 21  |20 ..          12|11  ..        4|3:2|

--  +---------+-----------+-----------------+---------------+---+---+

--  | 5       |           | 9               | 8             | 2 |   |

--  +---------+-----------+-----------------+---------------+---+---+

--  ^                     ^                 ^               ^- LINE_INDEX_SIZE

--  5 bits                9 bits            LINE_NUMBER_SIZE

--

-- Since bits 26 downto 21 are not included in the tag, there will be a 

-- 'mirror' effect in the cache. We have split the memory space 

-- into 32 separate blocks of 1MB which is obviously not enough but will do

-- for the initial tests.

-- In subsequen versions of the cache, the tag size needs to be enlarged AND 

-- some of the top bits might be omitted when they're not needed to implement 

-- the default memory map (namely bit 30 which is always '0').

--

--

-- @note3: Possible bug in Quartus-II and workaround

--

-- I had to put a 'dummy' mux between the cache line store and the CPU in order 

-- to get rid of a quirk in Quartus-II synthseizer (V9.0 build 235).

-- If we omit this extra dummy layer of logic the synth will fail to infer the 

-- tag table as a BRAM and will use logic fabric instead, crippling performance.

-- The mux is otherwise useless and hits performance badly, but so far I haven't

-- found any other way to overcome this bug, not even with the helop of the  

-- Altera support forum.

--

-- @note4: Startup values for the cache tables

-- 

-- The cache tables has been given startup values; these are only for simulation

-- convenience and have no effect on the cache behaviour (and obviuosly they

-- are only used after FPGA config, not after reset). 

--

--------------------------------------------------------------------------------

-- This module interfaces the CPU to the following:

--

--  1.- Internal 32-bit-wide BRAM for read only

--  2.- Internal 32-bit I/O bus

--  3.- External 16-bit or 8-bit wide static memory (SRAM or FLASH)

--  4.- External 16-bit wide SDRAM (NOT IMPLEMENTED YET)

--

-- The SRAM memory interface signals are meant to connect directly to FPGA pins

-- and all outputs are registered (tco should be minimal).

-- SRAM data inputs are NOT registered, though. They go through a couple muxes

-- before reaching the first register so watch out for tsetup.

--

--------------------------------------------------------------------------------

-- External FPGA signals

--

-- This module has signals meant to connect directly to FPGA pins: the SRAM

-- interface. They are either direct register outputs or at most with an

-- intervening 2-mux, in order to minimize the Tco (clock-to-output).

--

-- The Tco of these signals has to be accounted for in the real SRAM interface.

-- For example, under Quartus-2 and with a Cyclone-2 grade -7 device, the

-- worst Tco for the SRAM data pins is below 5 ns, enough to use a 10ns SRAM

-- with a 20 ns clock cycle.

-- Anyway, you need to take care of this yourself (synthesis constraints).

--

--------------------------------------------------------------------------------

-- Interface to CPU

--

-- 1.- All signals coming from the CPU are registered.

-- 2.- All CPU inputs come directly from a register, or at most have a 2-mux in

--     between.

--

-- This means this block will not degrade the timing performance of the system,

-- as long as its logic is shallower than the current bottleneck (the ALU).

--

--------------------------------------------------------------------------------

-- KNOWN PROBLEMS:

--

-- 1.- All parameters hardcoded -- generics are almost ignored.

-- 2.- SRAM read state machine does not guarantee internal FPGA Thold. 

--     Currently it works because the FPGA hold tines (including an input mux

--     in the parent module) are far smaller than the SRAM response times, but

--     it would be better to insert an extra cycle after the wait states in

--     the sram read state machine.

--------------------------------------------------------------------------------

library ieee;

use ieee.std_logic_1164.all;

use ieee.std_logic_arith.all;

use ieee.std_logic_unsigned.all;

use work.mips_pkg.all;

entity mips_cache is

    generic (

        BRAM_ADDR_SIZE : integer    := 10;  -- BRAM address size

        SRAM_ADDR_SIZE : integer    := 17;  -- Static RAM/Flash address size

        -- these cache parameters are unused in this implementation, they're

        -- here for compatibility to the final cache module.

        LINE_SIZE : integer         := 4;   -- Line size in words

        CACHE_SIZE : integer        := 256  -- I- and D- cache size in lines

    );

    port(

        clk             : in std_logic;

        reset           : in std_logic;

        -- Interface to CPU core

        data_addr       : in std_logic_vector(31 downto 0);

        data_rd         : out std_logic_vector(31 downto 0);

        data_rd_vma     : in std_logic;

        code_rd_addr    : in std_logic_vector(31 downto 2);

        code_rd         : out std_logic_vector(31 downto 0);

        code_rd_vma     : in std_logic;

        byte_we         : in std_logic_vector(3 downto 0);

        data_wr         : in std_logic_vector(31 downto 0);

        mem_wait        : out std_logic;

        cache_enable    : in std_logic;

        ic_invalidate   : in std_logic;

        unmapped        : out std_logic;

        -- interface to FPGA i/o devices

        io_rd_data      : in std_logic_vector(31 downto 0);

        io_rd_addr      : out std_logic_vector(31 downto 2);

        io_wr_addr      : out std_logic_vector(31 downto 2);

        io_wr_data      : out std_logic_vector(31 downto 0);

        io_rd_vma       : out std_logic;

        io_byte_we      : out std_logic_vector(3 downto 0);

        -- interface to synchronous 32-bit-wide FPGA BRAM (possibly used as ROM)

        bram_rd_data    : in std_logic_vector(31 downto 0);

        bram_wr_data    : out std_logic_vector(31 downto 0);

        bram_rd_addr    : out std_logic_vector(BRAM_ADDR_SIZE+1 downto 2);

        bram_wr_addr    : out std_logic_vector(BRAM_ADDR_SIZE+1 downto 2);

        bram_byte_we    : out std_logic_vector(3 downto 0);

        bram_data_rd_vma: out std_logic;

        -- interface to asynchronous 16-bit-wide or 8-bit-wide static memory

        sram_address    : out std_logic_vector(SRAM_ADDR_SIZE-1 downto 0);

        sram_data_rd    : in std_logic_vector(15 downto 0);

        sram_data_wr    : out std_logic_vector(15 downto 0);

        sram_byte_we_n  : out std_logic_vector(1 downto 0);

        sram_oe_n       : out std_logic

    );

end entity mips_cache;

architecture direct of mips_cache is

-- Address of line within line store

constant LINE_NUMBER_SIZE : integer := log2(CACHE_SIZE);

-- Address of word within line

constant LINE_INDEX_SIZE : integer  := log2(LINE_SIZE);

-- Address of word within line store

constant LINE_ADDR_SIZE : integer   := LINE_NUMBER_SIZE+LINE_INDEX_SIZE;

-- Code tag size, excluding valid bit

-- FIXME should be a generic

constant CODE_TAG_SIZE : integer    := 14;

-- Data tag size, excluding valid bit

-- FIXME should be a generic

constant DATA_TAG_SIZE : integer    := 14;

-- Wait state counter -- we're supporting static memory from 10 to >100 ns

-- (0 to 7 wait states with realistic clock rates).

subtype t_wait_state_counter is std_logic_vector(2 downto 0);

-- State machine ----------------------------------------------------

type t_cache_state is (

    idle,                       -- Cache is hitting, control machine idle

    -- Code refill --------------------------------------------------

    code_refill_bram_0,         -- pc in bram_rd_addr

    code_refill_bram_1,         -- op in bram_rd

    code_refill_bram_2,         -- op in code_rd

    code_refill_sram_0,         -- rd addr in SRAM addr bus (low hword)

    code_refill_sram_1,         -- rd addr in SRAM addr bus (high hword)

    code_refill_sram8_0,        -- rd addr in SRAM addr bus (byte 0)

    code_refill_sram8_1,        -- rd addr in SRAM addr bus (byte 1)

    code_refill_sram8_2,        -- rd addr in SRAM addr bus (byte 2)

    code_refill_sram8_3,        -- rd addr in SRAM addr bus (byte 3)

    code_crash,                 -- tried to run from i/o or something like that

    -- Data refill & write-through ----------------------------------

    data_refill_sram_0,         -- rd addr in SRAM addr bus (low hword)

    data_refill_sram_1,         -- rd addr in SRAM addr bus (high hword)

    data_refill_sram8_0,        -- rd addr in SRAM addr bus (byte 0)

    data_refill_sram8_1,        -- rd addr in SRAM addr bus (byte 1)

    data_refill_sram8_2,        -- rd addr in SRAM addr bus (byte 2)

    data_refill_sram8_3,        -- rd addr in SRAM addr bus (byte 3)

    data_refill_bram_0,         -- rd addr in bram_rd_addr

    data_refill_bram_1,         -- rd data in bram_rd_data

    data_refill_bram_2,

    data_read_io_0,             -- rd addr on io_rd_addr, io_vma active

    data_read_io_1,             -- rd data on io_rd_data

    data_write_io_0,            -- wr addr & data in io_wr_*, io_byte_we active

    data_writethrough_sram_0a,  -- wr addr & data in SRAM buses (low hword)

    data_writethrough_sram_0b,  -- WE asserted

    data_writethrough_sram_0c,  -- WE deasserted

    data_writethrough_sram_1a,  -- wr addr & data in SRAM buses (high hword)

    data_writethrough_sram_1b,  -- WE asserted

    data_writethrough_sram_1c,  -- WE deasserted

    data_ignore_write,          -- hook for raising error flag FIXME untested

    data_ignore_read,           -- hook for raising error flag FIXME untested

    -- Other states -------------------------------------------------

    --code_wait_for_dcache,       -- wait for D-cache to stop using the buses

    bug                         -- caught an error in the state machine

   );

-- Cache state machine state register & next state

signal ps, ns :             t_cache_state;

-- Wait state down-counter, formally part of the state machine register

signal ws_ctr :             t_wait_state_counter;

-- Wait states for memory being accessed

signal ws_value :           t_wait_state_counter;

-- Asserted to initialize the wait state counter

signal load_ws_ctr :        std_logic;

-- Asserted when the wait state counter has reached zero

signal ws_wait_done :       std_logic;

-- Refill word counters

signal code_refill_ctr :    integer range 0 to LINE_SIZE-1;

signal data_refill_ctr :    integer range 0 to LINE_SIZE-1;

signal data_refill_start :  std_logic;

signal data_refill_end :    std_logic;

-- CPU interface registers ------------------------------------------

-- Registered CPU addresses

signal data_rd_addr_reg :   t_pc;

signal data_wr_addr_reg :   t_pc;

signal code_rd_addr_reg :   t_pc;

-- Data write register (data to be written to external RAM)

signal data_wr_reg :        std_logic_vector(31 downto 0);

-- Registered byte_we vector

signal byte_we_reg :        std_logic_vector(3 downto 0);

-- SRAM interface ---------------------------------------------------

-- Stores first (high) Half-Word read from SRAM

signal sram_rd_data_reg :   std_logic_vector(31 downto 8);

-- Data read from SRAM, valid in refill_1

signal sram_rd_data :       t_word;

-- I-cache ----------------------------------------------------------

subtype t_line_addr is std_logic_vector(LINE_NUMBER_SIZE-1 downto 0);

subtype t_word_addr is std_logic_vector(LINE_ADDR_SIZE-1 downto 0);

subtype t_code_tag is std_logic_vector(CODE_TAG_SIZE+1-1 downto 0);

type t_code_tag_table is array(CACHE_SIZE-1 downto 0) of t_code_tag;

type t_code_line_table is array((CACHE_SIZE*LINE_SIZE)-1 downto 0) of t_word;

-- Code tag table (stores line tags) (@note4)

signal code_tag_table :     t_code_tag_table   := (others => "000000000000000");

-- Code line table  (stores lines)

signal code_line_table :    t_code_line_table  := (others => X"00000000");

-- Tag from code fetch address ('target' address, straight from CPU lines)

signal code_tag :           t_code_tag;

-- Registered code_tag, used matching after reading from code_tag_table

signal code_tag_reg :       t_code_tag;

-- Tag read from cache (will be matched against code_tag_reg)

signal code_cache_tag :     t_code_tag;

-- Code cache line address for read and write ports

signal code_line_addr :     t_line_addr;

-- Code cache word address (read from cache)

signal code_word_addr :     t_word_addr;

-- Code cache word address (write to cache in refills)

signal code_word_addr_wr :  t_word_addr;

-- Word written into code cache

signal code_refill_data :   t_word;

-- Address the code refill data is fetched from

signal code_refill_addr :   t_pc;

-- code word read from cache

signal code_cache_rd :      t_word;

-- raised when code_cache_rd is not valid due to a cache miss

signal code_miss :          std_logic;

-- code_miss for accesses to CACHED areas with cache enabled

signal code_miss_cached : std_logic;

-- code_miss for accesses to UNCACHED areas OR with cache disabled

signal code_miss_uncached : std_logic;

-- '1' when the I-cache state machine stalls the pipeline (mem_wait)

signal code_wait :          std_logic;

-- D-cache ----------------------------------------------------------

subtype t_data_tag is std_logic_vector(DATA_TAG_SIZE+1-1 downto 0);

type t_data_tag_table is array(CACHE_SIZE-1 downto 0) of t_data_tag;

type t_data_line_table is array((CACHE_SIZE*LINE_SIZE)-1 downto 0) of t_word;

-- Data tag table (stores line tags)

signal data_tag_table :     t_data_tag_table   := (others => "000000000000000");

-- Data line table  (stores lines)

signal data_line_table :    t_data_line_table  := (others => X"00000000");

-- Asserted when the D-Cache line table is to be written to

signal update_data_line :   std_logic;

signal update_data_tag :    std_logic;

-- Tag from data load address ('target' address, straight from CPU lines)

signal data_tag :           t_data_tag;

-- Registered data_tag, used matching after reading from data_tag_table

signal data_tag_reg :       t_data_tag;

-- Tag read from cache (will be matched against data_tag_reg)

signal data_cache_tag :     t_data_tag;

-- '1' when the read OR write data address tag matches the cache tag

signal data_tags_match :    std_logic;

-- Data cache line address for read and write ports

signal data_line_addr :     t_line_addr;

-- Data cache word address (read from cache)

signal data_word_addr :     t_word_addr;

-- Data cache word address (write to cache in refills)

signal data_word_addr_wr :  t_word_addr;

-- Word written into data cache

signal data_refill_data :   t_word;

-- Address the code refill data is fetched from (word address)

signal data_refill_addr :   t_pc;

-- Data word read from cache

signal data_cache_rd :      t_word;

-- Raised when data_cache_rd is not valid due to a cache miss

signal data_miss :          std_logic;

-- Data miss logic, portion used with cache enabledº

signal data_miss_cached :   std_logic;

-- Data miss logic, portion used with cach disabled

signal data_miss_uncached : std_logic;

-- Active when LW follows right after a SW (see caveats in code below)

signal data_miss_by_invalidation : std_logic;

-- Active when the data tag comparison result is valid (1 cycle after rd_vma)

-- Note: no relation to byte_we. 

signal data_tag_match_valid:std_logic;

-- Active when the D-cache state machine stalls the pipeline (mem_wait)

signal data_wait :          std_logic;

-- Active when there's a write waiting to be done

signal write_pending :      std_logic;

-- Active when there's a read waiting to be done

signal read_pending :       std_logic;

-- Address decoding -------------------------------------------------

-- Address slices used to decode

signal code_rd_addr_mask :  t_addr_decode;

signal data_rd_addr_mask :  t_addr_decode;

signal data_wr_addr_mask :  t_addr_decode;

-- Memory map area being accessed for each of the 3 buses:

signal code_rd_attr :       t_range_attr;

signal data_rd_attr :       t_range_attr;

signal data_wr_attr :       t_range_attr;

--------------------------------------------------------------------------------

begin

--------------------------------------------------------------------------------

-- Cache control state machine

cache_state_machine_reg:

process(clk)

begin

   if clk'event and clk='1' then

        if reset='1' then

            ps <= idle;

        else

            ps <= ns;

        end if;

    end if;

end process cache_state_machine_reg;

-- Unified control state machine for I-Cache and D-cache -----------------------

-- FIXME The state machine deals with all supported widths and types of memory, 

-- there should be a simpler version with only SRAM/ROM and DRAM.

control_state_machine_transitions:

process(ps, code_rd_vma, data_rd_vma, code_miss,

        data_wr_attr.mem_type, data_rd_attr.mem_type, code_rd_attr.mem_type,

        ws_wait_done, code_refill_ctr, data_refill_ctr,

        write_pending, read_pending)

begin

    case ps is

    when idle =>

        if code_miss='1' then

            case code_rd_attr.mem_type is

            when MT_BRAM        => ns <= code_refill_bram_0;

            when MT_SRAM_16B    => ns <= code_refill_sram_0;

            when MT_SRAM_8B     => ns <= code_refill_sram8_0;

            when others         => ns <= code_crash;

            end case;

        elsif write_pending='1' then

            case data_wr_attr.mem_type is

            when MT_BRAM        => ns <= data_ignore_write;

            when MT_SRAM_16B    => ns <= data_writethrough_sram_0a;

            when MT_IO_SYNC     => ns <= data_write_io_0;

            -- FIXME ignore write to undecoded area (clear pending flag)

            when others         => ns <= data_ignore_write;

            end case;

        elsif read_pending='1' then

            case data_rd_attr.mem_type is

            when MT_BRAM        => ns <= data_refill_bram_0;

            when MT_SRAM_16B    => ns <= data_refill_sram_0;

            when MT_SRAM_8B     => ns <= data_refill_sram8_0;

            when MT_IO_SYNC     => ns <= data_read_io_0;

            -- FIXME ignore read from undecoded area (clear pending flag)

            when others         => ns <= data_ignore_read;

            end case;

        else

            ns <= ps;

        end if;

    -- Code refill states -------------------------------------------

    when code_refill_bram_0 =>

        ns <= code_refill_bram_1;

    when code_refill_bram_1 =>

        ns <= code_refill_bram_2;

    when code_refill_bram_2 =>

        if code_refill_ctr/=0 then

            -- Still not finished refilling line, go for next word

            ns <= code_refill_bram_0;

        else

            -- If there's a data operation pending, do it now

            if write_pending='1' then

                case data_wr_attr.mem_type is

                when MT_BRAM        => ns <= data_ignore_write;

                when MT_SRAM_16B    => ns <= data_writethrough_sram_0a;

                when MT_IO_SYNC     => ns <= data_write_io_0;

                -- FIXME ignore write to undecoded area (clear pending flag)

                when others         => ns <= data_ignore_write;

                end case;

            elsif read_pending='1' then

                case data_rd_attr.mem_type is

                when MT_BRAM        => ns <= data_refill_bram_0;

                when MT_SRAM_16B    => ns <= data_refill_sram_0;

                when MT_SRAM_8B     => ns <= data_refill_sram8_0;

                when MT_IO_SYNC     => ns <= data_read_io_0;

                -- FIXME ignore read from undecoded area (clear pending flag)

                when others         => ns <= data_ignore_read;

                end case;

            else

                ns <= idle;

            end if;

        end if;

    when code_refill_sram_0 =>

        if ws_wait_done='1' then

            ns <= code_refill_sram_1;

        else

            ns <= ps;

        end if;

    when code_refill_sram_1 =>

        if code_refill_ctr/=0 and ws_wait_done='1' then

            -- Still not finished refilling line, go for next word

            ns <= code_refill_sram_0;

        else

            if ws_wait_done='1' then

                -- If there's a data operation pending, do it now

                if write_pending='1' then

                    case data_wr_attr.mem_type is

                    when MT_BRAM        => ns <= data_ignore_write;

                    when MT_SRAM_16B    => ns <= data_writethrough_sram_0a;

                    when MT_IO_SYNC     => ns <= data_write_io_0;

                    -- FIXME ignore write to undecoded area (clear pending flag)

                    when others         => ns <= data_ignore_write;

                    end case;

                elsif read_pending='1' then

                    case data_rd_attr.mem_type is

                    when MT_BRAM        => ns <= data_refill_bram_0;

                    when MT_SRAM_16B    => ns <= data_refill_sram_0;

                    when MT_SRAM_8B     => ns <= data_refill_sram8_0;

                    when MT_IO_SYNC     => ns <= data_read_io_0;

                    -- FIXME ignore read from undecoded area (clear pending flag)

                    when others         => ns <= data_ignore_read;

                    end case;

                else

                    ns <= idle;

                end if;

            else

                ns <= ps;

            end if;

        end if;

    when code_refill_sram8_0 =>

        if ws_wait_done='1' then

            ns <= code_refill_sram8_1;

        else

            ns <= ps;

        end if;

    when code_refill_sram8_1 =>

        if ws_wait_done='1' then

            ns <= code_refill_sram8_2;

        else

            ns <= ps;

        end if;

    when code_refill_sram8_2 =>

        if ws_wait_done='1' then

            ns <= code_refill_sram8_3;

        else

            ns <= ps;

        end if;

    when code_refill_sram8_3 =>

        if code_refill_ctr/=0 and ws_wait_done='1' then

            -- Still not finished refilling line, go for next word

            ns <= code_refill_sram8_0;

        else

            if ws_wait_done='1' then

                -- If there's a data operation pending, do it now

                if write_pending='1' then

                    case data_wr_attr.mem_type is

                    when MT_BRAM        => ns <= data_ignore_write;

                    when MT_SRAM_16B    => ns <= data_writethrough_sram_0a;

                    when MT_IO_SYNC     => ns <= data_write_io_0;

                    -- FIXME ignore write to undecoded area (clear pending flag)

                    when others         => ns <= data_ignore_write;

                    end case;

                elsif read_pending='1' then

                    case data_rd_attr.mem_type is

                    when MT_BRAM        => ns <= data_refill_bram_0;

                    when MT_SRAM_16B    => ns <= data_refill_sram_0;

                    when MT_SRAM_8B     => ns <= data_refill_sram8_0;

                    when MT_IO_SYNC     => ns <= data_read_io_0;

                    -- FIXME ignore read from undecoded area (clear pending flag)

                    when others         => ns <= data_ignore_read;

                    end case;

                else

                    ns <= idle;

                end if;

            else

                ns <= ps;

            end if;

        end if;

    -- Data refill & write-through states ---------------------------

    when data_write_io_0 =>

        ns <= idle;

    when data_read_io_0 =>

        ns <= data_read_io_1;

    when data_read_io_1 =>

        ns <= idle;

    when data_refill_sram8_0 =>

        if ws_wait_done='1' then

            ns <= data_refill_sram8_1;

        else

            ns <= ps;

        end if;

    when data_refill_sram8_1 =>

        if ws_wait_done='1' then

            ns <= data_refill_sram8_2;

        else

            ns <= ps;

        end if;

    when data_refill_sram8_2 =>

        if ws_wait_done='1' then

            ns <= data_refill_sram8_3;

        else

            ns <= ps;

        end if;

    when data_refill_sram8_3 =>

        if ws_wait_done='1' then

            if data_refill_ctr/=LINE_SIZE-1 then

                ns <= data_refill_sram8_0;

            else

                ns <= idle;

            end if;

        else

            ns <= ps;

        end if;

    when data_refill_sram_0 =>

        if ws_wait_done='1' then

            ns <= data_refill_sram_1;

        else

            ns <= ps;

        end if;

    when data_refill_sram_1 =>

        if ws_wait_done='1' then

            if data_refill_ctr=LINE_SIZE-1 then

                ns <= idle;

            else

                ns <= data_refill_sram_0;

            end if;

        else

            ns <= ps;

        end if;

    when data_refill_bram_0 =>

        ns <= data_refill_bram_1;

    when data_refill_bram_1 =>

        ns <= data_refill_bram_2;

    when data_refill_bram_2 =>

        if data_refill_ctr/=(LINE_SIZE-1) then

            -- Still not finished refilling line, go for next word

            ns <= data_refill_bram_0;

        else

            if read_pending='1' then

                case data_rd_attr.mem_type is

                when MT_BRAM        => ns <= data_refill_bram_0;

                when MT_SRAM_16B    => ns <= data_refill_sram_0;

                when MT_SRAM_8B     => ns <= data_refill_sram8_0;